Cargo API

[rhyzx]

Here is a new API called cargo, it is like queue with single worker

queue with concurrency 2

cargo with payload 2

[caolan]

Wow, I've never had a pull request with animated gifs in it before! A+ for effort :)

Can you give me an example use case for this?

[indexzero]

@caolan Because of lack of proper tail recursion in Javascript, async.forEach (and I'm guessing async.queue) methods will crash with StackOverflow errors with extremely large arrays.

I'm encountered this doing something like

  async.forEachSeries(new Array(5000), function (i, next) {
    process.nextTick(next);
  }, function (err) {
    console.log('done');
  });

[CMCDragonkai]

What would I be able to do that I couldn't do with this cargo method?

[caolan]

@indexzero you shouldn't get a StackOverflow with that code. I just ran it with new Array(1000000) to make sure I wasn't going crazy ;)

[rhyzx]

@caolan I see. Try my best effort to explain it with my bad english _(:3｣ )=

Cargo is not very common, but it is very useful in some situation

Example

Upload(ajax) data with cargo

// Every moment there is only one ajax in the operation
// This can save the largest degree of server requests and keep the instant of data
var dataUploader = async.cargo(function (ids, callback) {
    $.post('/userAddItem', ids, callback);
}, 500); //max 500 data per request

// user add item event
$(document).on('click', '.item', function () {
    var itemName = this.getAttribute('data-item-name');
    var itemId = this.getAttribute('data-item-id');

    dataUploader.push(itemId, function () {
        alert('add ' +itemName +' success!');
    });
});

Queue version

var dataToUpload = [];
var dataUploader = async.queue(function (nil, nonUse) {
    var data = dataToUpload.splice(0, 500); //max 500
    var ids = [];
    var callbacks = [];

    for (var i=0, len=data.length; i<len; i++) {
        var ret = data[i];
        ids.push(ret.itemId);
        callbacks.push(ret.callback);
    }

    $.post('/userAddItem', ids, function () {
        for (var i=0, len=callbacks.length; i<len; i++) {
            callbacks[i]();//exec callback
        }
    });
}, 1); // single worker

// user add item event
$(document).on('click', '.item', function () {
    var itemName = this.getAttribute('data-item-name');
    var itemId = this.getAttribute('data-item-id');

    dataToUpload.push({
        itemId: itemId,
        callback: function () {
            console.log('add ' +itemName +' success!');
        }
    });

    if (dataUploader.length() < 1) {
        //call uploader to process data
        dataUploader.push(0);
    }
});

Example2(node)

This example may not be appropriate.

Sync files into one file by watch method

// dataWriter will wirte result when files updated
var dataMap = {};
var dataWriter = async.cargo(function (tasks, callback) {
    //set updated data
    tasks.forEach(function (v, i) {
        dataMap[v.fileName] = v.fileData;
    });

    //concat data include new&old
    var result = JSON.stringify(dataMap);

    fs.writeFile('result', result, callback);
}); // ulimited payload

fs.watchFile('source', function () {
    //file updated
    fs.readFile('source', function (err, data) {
        dataWriter.push({
            fileName: 'source',
            fileData: data
        });
    });
});
fs.watchFile('source2', function () {
    //file updated
    fs.readFile('source2', function (err, data) {
        dataWriter.push({
            fileName: 'source2',
            fileData: data
        });
    });
});
//...

Standard version

var dataMap = {};

var isWriting = false;
var writeAgain = false;
function writeData() {
    if (isWriting) return writeAgain = true;

    isWriting = true;
    writeAgain = false;

    var result = JSON.stringify(dataMap);
    fs.writeFile('result', result, function () {
        isWriting = false;

        //check whether dataMap is changed
        if (writeAgain) {
            writeData();
        }
    });
}

fs.watchFile('source', function () {
    //file updated
    fs.readFile('source', function (err, data) {
        dataMap['source'] = data;
        writeData(); //process
    });
});
fs.watchFile('source2', function () {
    //file updated
    fs.readFile('source2', function (err, data) {
        dataMap['source2'] = data;
        writeData();
    });
});
//...

[DTrejo]

seems useful, also great illustration :)

[RubyTuesdayDONO]

that animation is awesome! do you have more? they would be great to include on the readme (either embedded or linked). i had some trouble understanding cargo vs. queue - until i came across this jewel. instant comprehension! 👍

[FGRibreau]

Same here, thanks for the gif @rhyzx !

[brianmaissy]

+1 to embed or link this animations in the readme

[dandv]

If a picture speaks a thousand words, and animation should be included in the README.