cartodb4/node-pg-copy-streams
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
67 Commits 5 Branches 14 Tags 181 KiB
61bc713e0c
Commit Graph

67 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rafa de la Torre
61bc713e0c Improve performance of COPY TO #56 
Under some circumstances, the COPY TO streamming can be CPU-bound,
particularly when PG holds the resultset in memory buffers and the size
of the rows << chunk (64 KB in my linux box).

This commits improves the situation by creating a buffer of `chunk`
size and fitting in as many rows as it can before pushing them. This
results in more balanced read and writes (in terms of size and in bigger
chunks) as well as more frequent calls to the callback, thus freeing the
main loop for other events to be processed, and therefore avoiding
starvation.
 2018-06-08 15:04:42 +02:00
jeromew
e15feb199a README.md: copy-from error and vacuum #26 2016-08-23 12:10:25 +02:00
jeromew
191a4ec16a Fix documentation of copy-from completion 2016-08-23 11:32:10 +02:00
jeromew
399bff7ed7 Bump version 2016-08-22 16:27:04 +00:00
jeromew
8174e10fb5 Merge pull request #54 from jeromew/upstream-end 
Test: `end` event should not be triggered 2 times on copy-from
 2016-08-22 17:47:29 +02:00
jeromew
f29aef3bba Merge pull request #53 from jeromew/upstream-frontier 
Bugfix - wrong tests on chunk frontiers
 2016-08-22 17:47:01 +02:00
jeromew
b2e108571e Issue #54: We should probably delay the _flush cb() to CommandComplete 2016-07-30 00:08:54 +00:00
jeromew
7003f6070f Fix issue #54: end is being triggered 2 times 2016-07-29 23:51:41 +00:00
jeromew
ade7ab95a0 Test: end event should not be triggered 2 times on copy-from 2016-07-29 23:47:39 +00:00
jeromew
9ccda04036 Bugfix - Chunk frontiers were not correctly tested 2016-07-28 23:05:50 +00:00
jeromew
a5e532f20b Bugfix - wrong tests on chunk frontiers 2016-07-28 22:55:02 +00:00
jeromew
2a4db2920e Fix NoticeResponse test and handle other messages (#52) 2016-07-28 16:13:22 -05:00
jeromew
f155899570 Add a test for NoticeResponse handling (#51) 2016-07-28 13:15:59 -05:00
brianc
18be125596 Bump version 2016-07-26 14:40:01 -05:00
jeromew
ae5b344395 Refactor message format codes handling (#45) 2016-07-26 14:39:48 -05:00
jeromew
ee84aba89f Use end stream option instead of not calling _flush callback (#44) 2016-07-26 14:39:33 -05:00
brianc
e0aa7db324 Bump version 2016-05-24 17:20:26 -05:00
Jonathan Bergknoff
0f0ddf7ad4 Expose row count after COPY FROM command (#37) 
* Expose row count after COPY FROM command

* correct conditional
 2016-05-24 17:20:16 -05:00
brianc
ed57e131e9 Drop support for node@v0.10 2016-05-03 13:21:28 -05:00
Brian C
c4a0e6dd58 Fix compatibility with newer versions of node (#39) 
* Eliminate detach/reattach strategy as it isn't able to differentiate between on/once and inconsistenly loses unshifted data depending on node version.  Instead just split stream and send it to copy stream and the connection.stream at the same time.  Disconnecting copy stream just means unpiping.  Added handleCopyData to fulfill query contract but ignore the incoming data.

Add node 4.2.2 to Travis
Minimum postgres 9.2 to allow tests to complete in Travis

Remove test that is no longer needed since we no longer disconnect/reconnect listeners

* Add resume

* Remove node 0.10 and add 0.12

* Re-enable old tests

* Add more versions to the travis test matrix
 2016-05-03 13:20:04 -05:00
Dan Robinson
d5b5c8c569 Merge pull request #33 from alexconlin/master 
Change file format in example from .tdv to .tsv
 2015-12-09 17:14:26 -08:00
Alex Conlin
6981ea6ac5 Change file format in example from .tdv to .tsv 2015-12-09 20:31:54 +00:00
brianc
d2b9677a02 Fix travis build 2015-03-09 08:11:33 -06:00
Brian M. Carlson
bfee86543f Remove database from .travis.yml - that is not the problem 2014-09-16 00:34:34 -04:00
Brian C
21f47220fc Merge pull request #20 from drob/error-message 
Better test of error handling after initial response.
 2014-09-16 00:33:21 -04:00
Dan
b1b613125f Better test of error handling after initial response. 2014-09-16 00:24:14 -04:00
Brian M. Carlson
4222053744 Last try before I abandon travis 2014-09-15 23:50:14 -04:00
Brian M. Carlson
040ed5f4da Mess with travis config 2014-09-15 23:48:14 -04:00
Brian M. Carlson
f9ee1c083a Merge branch 'master' of github.com:brianc/node-pg-copy-streams 2014-09-15 23:39:49 -04:00
Brian M. Carlson
dcfffd0670 Try new travis changes 2014-09-15 23:39:42 -04:00
Brian C
12e4ca33b0 Update README.md 
dat travis badge
 2014-09-15 21:10:42 -04:00
Brian M. Carlson
8f2355e454 0.3.0 2014-09-15 20:56:39 -04:00
Brian M. Carlson
d6eab36b66 Make tests a bit more robusto 2014-09-15 20:56:34 -04:00
Brian M. Carlson
33f6ecc11b Add workflow boilerplate files 2014-09-15 20:49:09 -04:00
Brian C
36572a8b7b Merge pull request #16 from drob/fix-docs 
Fixes pipe from a file to table example in README.md.
 2014-09-15 20:48:23 -04:00
Brian C
0c5d08edae Merge pull request #19 from drob/transform-opts 
Accept stream options in constructors, pass to internal transform streams.
 2014-09-15 20:47:38 -04:00
Dan
b78a3eb845 Accept stream options in constructors, pass to internal transform streams. 
Includes tests.
 2014-09-15 15:01:39 -04:00
Dan
107f007249 Fixes pipe from a file to table example in README.md. 2014-08-10 13:33:33 -07:00
Brian C
25b8d6da5f Update README.md 
Providing clarity for #6
 2014-05-01 11:07:33 -05:00
Dan Robinson
1c0c8871c1 Bump version 2014-04-07 11:56:28 -07:00
Dan Robinson
beb54334e2 Merge pull request #12 from drob/error-handling 
Adds handling for errors after initial response.
 2014-04-07 11:55:36 -07:00
Dan Robinson
1db9b3ec3d Adds handling for errors after initial response. 
Includes a test.
 2014-04-06 22:04:20 -07:00
Brian M. Carlson
1822f399d2 Bump version 2014-04-06 11:09:28 -05:00
Brian C
208861d057 Merge pull request #11 from drob/add-test 
Adds a smoke test for copying with BINARY.
 2014-04-06 11:06:16 -05:00
Brian C
1957554301 Merge pull request #10 from drob/handler-leak 
Fixes another event handler leak.
 2014-04-06 11:06:09 -05:00
Dan Robinson
bef3ba5fcd Adds a smoke test for copying with BINARY. 2014-04-06 04:50:54 -07:00
Dan Robinson
591a11c955 Fixes another event handler leak. 
It turns out 'error' handlers were leaking as well, although more slowly.
 2014-04-06 03:49:25 -07:00
Brian M. Carlson
b40918ddb8 Bump version 2014-03-30 00:10:18 -04:00
Brian C
b8946265c3 Merge pull request #8 from drob/handler-leak 
Fixes an event handler leak.
 2014-03-30 00:06:42 -04:00
Dan Robinson
d649905dbb Fixes an event handler leak. 
Includes a test. Also includes async as a devDependency.
 2014-03-28 04:19:19 -07:00