-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
5 second batching seems to only allow for messages every 5 seconds #10
Comments
Should it be a param instead? |
I think that it 'should' batch them, but should queue/coalesce for the From: Vincent Serpoul [email protected] Should it be a param instead? |
What if we use a channel and include a timer that processes the batch every 5 seconds? pointChannel := make(chan entry *logrus.Entry)
func (hook *InfluxDBHook) Fire(entry *logrus.Entry) error {
pointChannel <- entry
...
...
...
return nil
}
...
...
...
func (hook *InfluxDBHook) processEntries() error {
L:
for {
select {
case pt, open := <-pointChannel:
if !open {
break L
}
hook.batch.AddPoint(pt)
// when the batch size gets too large, write to adapter
if len(hook.batch) >= MAX_SIZE {
processBatch(hook.batch)
}
case time.NewTicker(time.Millisecond * 5000):
processBatch(hook.batch)
default:
// do nothing
}
}
return nil
} |
@matthalladay: what you describe is what it is supposed to do, are you saying it doesn't "queue" points and then write multiple points? I've used this code for a while on my side and didn't see any dropped points. Maybe I didn't pay enough attention? @abramovic , I'm not sure it would solve the issue, if there is one. I'm not sure what the issue is though :) |
@matthalladay @vincentserpoul /// This should remove remove the batching
NewInfluxDBWithConfig(&Config{BatchInterval: 0}) |
I'm sorry I'm a bit of a newb when it comes to Go, but I know what was
From: Vincent Serpoul [email protected] Should it be a param instead? |
I don't think this writes to InfluxDB every 5 seconds. It looks like it writes to InfluxDB if it's been more than 5 seconds since the last time a logrus entry was collected. if hook.lastBatchUpdate.Add(hook.batchInterval).Before(time.Now()) ||
len(hook.batchP.Points()) > 200 {
err = hook.client.Write(hook.batchP)
if err != nil {
return fmt.Errorf("Fire: %v", err)
}
hook.lastBatchUpdate = time.Now()
hook.batchP = nil
} Example:
|
@matthalladay - yes, I think that's what's happening with the current batching. I just made a pull request that should take care of that for you. @vincentserpoul thoughts? #11 |
Nick, Thanks for making changes so quickly! There is another issue (not so much Changes: Implementation:
From: Nick [email protected] @matthalladay[github.com] - yes, I think that's what's happening with the |
The images didn't get attached. Can you upload the images to Github? #10 Most likely the default precision is being set to milliseconds. Are you using NewInfluxDBHook or NewWithClientInfluxDBHook? |
@matthalladay @abramovic I will implement the new updated lib and bring some feedback, most probably 👍 nice reactivity Nick! |
@matthalladay are you still having issues with the timestamps? |
@matthalladay - any update? Thanks! |
This might be the intended usage of the hook, but I intend to potentially log messages faster than every five seconds and definitely don't want them aggregated. I just changed 5 to 0 in the code, but wanted to let you know.
The text was updated successfully, but these errors were encountered: