Where do I get a package for GNU Parallel?

1

Это ваша лента, где вы будете проводить большую часть времени, получая мгновенные уведомления о том, что интересует именно вас.

Наведите на изображение профиля и нажмите кнопку чтения, чтобы перестать читать любую учетную запись.

Самый быстрый способ поделиться чьим-либо твитом с вашими читателями — ретвитнуть его. Нажмите значок со стрелочками, чтобы мгновенно сделать это.

Поделитесь своими мыслями о любом твите, просто ответив на него. Найдите тему, которая вам интересна, и вступайте в беседу.

Мгновенно узнавайте о том, что обсуждают люди прямо сейчас.

Читайте больше учетных записей, чтобы получать мгновенные обновления о том, что вас больше всего интересует.

Читайте самые последние беседы на любую тему, мгновенно появляющиеся в вашей ленте.

Следите за тем, как разворачиваются лучшие...

0 0
2

ML: Ole, GNU Parallel is a specialist tool for running multiple jobs at the command line at the same time. Why did you develop it, and what are your uses for it?

OT: I often get in the situation that I need to run a script on each line of a bunch of lines, so back in 2001 I made a wrapper script for make -j to run command lines in parallel. This was the first basic version of parallel that later became GNU Parallel. The full history of GNU Parallel is at http://www.gnu.org/s/parallel/history.html

Today I use GNU Parallel even for tasks that do not really need to be run in parallel, simply because of its ease of replacing arguments on the command line. Like emptying all tables in a database:

sql -n mysql:/// 'show tables' | parallel sql mysql:/// DELETE FROM {};

To me it has become a bit of a sport to see if the tasks I do can be done more efficiently using GNU Parallel. When you have gotten used to it, a lot of the once-off scripts can often be written...

0 0
3

I was reading the ImageNet tutorial for Caffe (a deep learning framework), in which they need to resize a large number of images. It struck me that they might not be aware of GNU Parallel, since it is a great tool for this task. I recommend it to any data scientist out there since it is so simple to use and like many other GNU tools, with good chance already installed on your computer. If not, run apt-get install parallel on Debian. It might suggest that you to install moreutils to get parallel, but this installs the wrong software (explanation).

In the writeup, it says that the author used his own MapReduce framework to do it, but it can also be done sequentially as:

for name in *.jpeg; do convert -resize 256x256\! $name $name done

Instead of this sequential approach, you can run it in parallel with even less typing:

parallel convert -resize 256x256\! {} {} ::: *.jpeg

GNU Parallel will insert each filename at {} to form a command. Multiple commands will...

0 0
4

I'm working with a related issue to GNU Parallel where we suspect a bug in the most recent GNU parallel release (How To Avoid SIGCHLD error In Bash Script That Uses GNU Parallel) and I would like to revert to an older version.

Originally, this is the code I used to install:

wd=$(mktemp -d)
wget -nc -P $wd ftp://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2

cd $wd
tar -xf parallel-latest.tar.bz2
cd parallel-*
./configure && make && make install

(Code found here: which cygwin package to get `parallel` command?)

This is the code I found to uninstall:

(wget pi.dk/3 -qO - || curl pi.dk/3/) | bash
cd parallel-20*/
make uninstall

(Code found here: How to uninstall GNU parallel?)

Would it be correct to uninstall and then install the older package? Or, is there a way to install the older package and specify which version I would like to use?

Your insight and support of this question is highly...

0 0
5

/** * The file upload file which allows users to upload files via the default HTML . * * @package Meta Box */ /** * File field class which uses HTML to upload file. */ class RWMB_File_Field extends RWMB_Field { /** * Enqueue scripts and styles. */ public static function admin_enqueue_scripts() { wp_enqueue_style( 'rwmb-file', RWMB_CSS_URL . 'file.css', array(), RWMB_VER ); wp_enqueue_script( 'rwmb-file', RWMB_JS_URL . 'file.js', array( 'jquery-ui-sortable' ), RWMB_VER, true ); self::localize_script( 'rwmb-file', 'rwmbFile', array( // Translators: %d is the number of files in singular form. 'maxFileUploadsSingle' => __( 'You may only upload maximum %d file', 'meta-box' ), // Translators: %d is the number of files in plural form. 'maxFileUploadsPlural' => __( 'You may only upload maximum %d files', 'meta-box' ), ) ); } /** * Add custom actions. */ public static function add_actions() { add_action( 'post_edit_form_tag', array( __CLASS__, 'post_edit_form_tag' ) ); ...

0 0
6
...
0 0
7

I have seen several questions about this topic, but I lack the ability to translate this to my specific problem. I have a for loop that loops through sub directories and then executes a .sh script on a compressed text file inside each directory. I want to parallelize this process, but I’m struggling to apply gnu parallel.

Here is my loop:

for d in ./*/ ; do (cd "$d" && script.sh); done

I understand I need to input a list into parallel, so i have been trying this:

ls -d */ | parallel cd && script.sh

While this appears to get started, I get an error when gzip tries to unzip one of the txt files inside the directory, saying the file does not exist:

gzip: *.txt.gz: No such file or directory

However, when I run the original for loop, I have no issues aside from it taking a century to finish. Also, I only get the gzip error once when using parallel, which is so weird considering I have over 1000 sub-directories.

My questions are:

How do I get...

0 0
8

GNU Parallel is a great tool for executing commands in parallel on one or more nodes. If you put all of your commands in a file named commands.txt you can have GNU Parallel dynamically distribute the commands across all of the nodes and cores that were requested by a pbs job.

Single-node Examples

You have a file named commands.txt containing a list of commands and want to run one command per core:

$ module load parallel $ parallel < commands.txt

GNU Parallel automatically identifes the number of cores on the node and runs one command per core. Use the –jobs option to specify a different number of concurrent commands:

$ module load parallel $ parallel --jobs 2 < commands.txt

You want to run the same command (FastQC) on many (fastq) files, running one command per core:

$ module load parallel $ module load fastqc $ find ~/fastqfolder -name *.fastq | parallel "fastqc {}"

You want to run the same command (wc -l) on many files, running one command per...

0 0
9

USEFUL UNIX COMMANDS

cancel cat file USEFUL UNIX COMMANDS cancel print requested with lp Display the file cat file1 file2 > files Combine file1 and file2 into files cat file1 >> file2 chgrp [options] newgroup files Append

More information

Thirty Useful Unix Commands

Leaflet U5 Thirty Useful Unix Commands Last revised April 1997 This leaflet contains basic information on thirty of the most frequently used Unix Commands. It is intended for Unix beginners who need a

More information

UNIX - USEFUL COMMANDS

UNIX - USEFUL COMMANDS http://www.tutorialspoint.com/unix/unix-useful-commands.htm Copyright tutorialspoint.com This quick guide lists commands, including a syntax and brief description. For more detail,

More information

UNIX Tutorial for Beginners

UNIX Tutorial for Beginners Introduction to the UNIX Operating System What is UNIX? Files and...

0 0
10

Article describing tool (for citations):

O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.

Author's website for obtaining code:

http://www.gnu.org/software/parallel/

All new computers have multiple cores. Many bioinformatics tools are serial in nature and will therefore not use the multiple cores. However, many bioinformatics tasks (especially within NGS) are extremely parallelizeable:

Run the same program on many files Run the same program on every sequence

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving...

0 0
11

ICU is a cross-platform Unicode based globalization library. It includes support for locale-sensitive string comparison, date/time/number/currency/message formatting, text boundary detection, character set conversion and so on.

There are many versions of compilers on so many platforms that we cannot build them all and guarantee compatibility between them all even on the same platform. Due to these restrictions, we only distribute a limited number of binary versions of ICU, but we will assist in building other versions from source.

You can use the Cygwin build environment to build ICU from source against the MSVC compiler. See the ICU4C Readme.

We can try.. make sure you read the latest "readme" and also the

section. You might also

, and then posting a question there. Additionally, sites such as

may have helpful tips for your topic.

What is the ICU binary compatibility policy?

Please see the section on binary compatibility...

0 0
12
Berkeley UPC downloads

Downloading and Installing Berkeley UPC

There are two main steps to get UPC running on your system using the Berkeley UPC Runtime. You will need to build from source on your system, and there are several ways to do this depending on what option(s) you want for UPC translation/compilation.

Required component: The Berkeley UPC Runtime/driver package

The Berkeley UPC Runtime is required by all configurations. Source Download: berkeley_upc-2.26.0.tar.gz (7.0MB)

MD5 Hash = d73f46c4f7a8d124781097e2bab28184

The runtime works on all our supported platforms.

The runtime distribution includes the GASNet portable networking library--if you wish to download GASNet separately, you can get it from the GASNet website.

INSTALL.TXT explains how to configure and build the Berkeley UPC runtime, including optional directions for configuring the 'upcc' driver to use any of the five UPC compiler options...

0 0
13
...
0 0
14
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58 program cartesian include 'mpif.h' integer SIZE, UP, DOWN, LEFT, RIGHT parameter(SIZE=16) parameter(UP=1) parameter(DOWN=2) parameter(LEFT=3) parameter(RIGHT=4) integer numtasks, rank, source, dest, outbuf, i, tag, ierr, & inbuf(4), nbrs(4), dims(2), coords(2), periods(2), reorder integer stats(MPI_STATUS_SIZE, 8), reqs(8) integer cartcomm ! required variable data inbuf /MPI_PROC_NULL,MPI_PROC_NULL,MPI_PROC_NULL,MPI_PROC_NULL/, & dims /4,4/, tag /1/, periods /0,0/, reorder /0/ call MPI_INIT(ierr) ...
0 0
15

GnuPG is a tool for secure communication. This chapter is a quick-start guide that covers the core functionality of GnuPG. This includes keypair creation, exchanging and verifying keys, encrypting and decrypting documents, and authenticating documents with digital signatures. It does not explain in detail the concepts behind public-key cryptography, encryption, and digital signatures. This is covered in Chapter 2. It also does not explain how to use GnuPG wisely. This is covered in Chapters 3 and 4.

GnuPG uses public-key cryptography so that users may communicate securely. In a public-key system, each user has a pair of keys consisting of a private key and a public key. A user's private key is kept secret; it need never be revealed. The public key may be given to anyone with whom the user wants to communicate. GnuPG uses a somewhat more sophisticated scheme in which a user has a primary keypair and then zero or more additional subordinate keypairs. The primary and...

0 0