Viewing Events with multi-line event payload is not ideal

[rauhs]

Hi all,

We have accumulated more and more of our custom ETW events over the year (they're great!). For us the payload of the events can be quite large. We often dump multi-line strings as event payloads.

The are tough to display: The export to Excel/CSV/XML does't show them well

What we have used is "Dump Event": That works okay: The XML attributes break the line on newlines.

However: The XML attribute is XML-Escaped. E.g. here is part of an SQL explain that is one of our events:

Predicate Information (identified by operation id):
    ---------------------------------------------------
     
       3 - filter("tbl0"."USERID" IS NULL AND "tbl0"."CLIENTID" IS NULL AND "tbl0"."USERGROUPID" IS NULL AND 
                  "tbl0"."ORGANIZATIONID" IS NULL OR "tbl0"."USERID" IS NULL AND "tbl0"."USERGROUPID"=11 OR "tbl0"."USERID" IS 
                  NULL AND "tbl0"."USERGROUPID" IS NULL AND "tbl0"."CLIENTID"=1 AND "tbl0"."ORGANIZATIONID" IS NULL OR 
                  "tbl0"."USERID"=10021517062 OR "tbl0"."USERID" IS NULL AND "tbl0"."ORGANIZATIONID"=10052802872 AND 
                  "tbl0"."USERGROUPID" IS NULL)

This is very tough to read. We have to manually copy that and unescape to just read it.

Ideas:

• If the payload was not an Attribute but an XML Tag: We could use CDATA and avoid the escaping.

Question:

1. Is the "dump event" considered some standard? I imagine we can't change from attributes to tags, right?
2. Any way we could extend this so we change the format? Maybe a second command (ALT+SHIFT+D?)?
3. Or just use a User Setting to change the Dump Event format?

Any other ideas? I'd certainly be open to send a pull request.

Thanks for reading!

[brianrob]

I don't think that the dump event concept follows a standard, or at least I'm not aware of one here. At the same time, I think it's going to be very hard to change this as there are lots of parsers out there that already implement ToXml with attributes for each field.

It sounds like the workflow you're employing is to open PerfView and do the export from there. I wonder if it might be worthwhile for you to create a TraceEvent-based application that pulls the events you care about and exports them as you'd like them. This would give you complete control of what things look like and would be relatively low-cost to produce. I bet your favorite LLM could probably produce it for you. :)

Would that work for you? I think that would give you control over time as well when your needs change, and anyone can do this, so others that have different needs would be free to do the same thing.

[rauhs]

Thanks for the quick reply!

It sounds like the workflow you're employing is to open PerfView and do the export from there.

We don't really export much from PerfView. Maybe every now and then we use the Excel export. But very rarely. I mentioned it above because I tried it out and it didn't show us the new lines....

Writing an exporter would not work for us: We certainly love all the other features the event viewer gives us (Best: ALT+S, show stack) and filtering etc etc.

I know that most people don't use multi-line payload for (their) ETW event: Though, I still think there might be value in PerfView supporting this a bit better.

Another idea: If I press CTRL+C on a cell in the event viewer and that event payload has at least one newline: Could we copy the event including the newlines?
Even though that would be a bit of change in behavior: I would imagine that most other PerfView users would likely see this as a positive change? What do you think?

[brianrob]

I would support having some better support for newlines. You're right that it's not super common, but you're definitely not the only one. I'd like to understand your workflow a bit better - do you tend to use PerfView directly, and specifically use the events viewer?

There is precedent to add optional behavior to PerfView and control it through the Options menu on the main window. This could be something that we put there. Then you could turn it on once and the change would persist across runs of PerfView.

[rauhs]

do you tend to use PerfView directly, and specifically use the events viewer?

We deploy a pretty large (5M LOC) Windows Forms app to our customers. It talks to their own locally running SQL DB.
The application can generate all kinds of SQL queries (completely dynamic) and also run all kinds of code depending on whatever is done. HTTP request. DB. Searching Lucene. Triggering PL/SQL. We even generate tables and columns dynamically and query these as well. We also have potentially 10-100 processes all on the same DB (servers etc).

In addition, we often do not have access to the system. As you can guess: We have a lot of performance problems, and this is where PerfView comes into play: We start it at the customer and then we analyze them.

Because of having our customer run the PerfView: We have added a lot of events over the years. Just so we can get to the bottom of the issue without a lot of pack and forth. It's great because we're very efficient in analyzing them. Nowadays even all logging (log4net) is in the PerfView trace. And of course, queries. And now: EXPLAIN PLAN (multiple lines).

We often do CPU analysis. Often Thread time. Sometimes memory analysis. Sometimes deadlocks.
We always check the event viewer unless it's a trivial CPU issue (which is getting less and less common as we fix our stuff).

Because we might have 20 processes running it's great to trace all of them at the same time and get a unified view of what's going on.

A normal trace (30-60s) will have tens of thousands of our events that we need to make sense of. This is where a multi-line viewer would be really nice.

There is precedent to add optional behavior to PerfView and control it through the Options menu on the main window.

That would be perfekt. Would that mean we change the output of <PrettyPrint>-part of the Dump? Or do you want CTRL-C to change the way? Or both?

[brianrob]

Would that mean we change the output of -part of the Dump? Or do you want CTRL-C to change the way? Or both?

We have options here. I could imagine you might just want to support a multi-line event viewer. I think that changing the Dump format is not likely to work because it's exposed outside of TraceEvent and anyone that implements a parser can implement it themselves. TraceParserGen implements it to match what's in TraceEvent.

What do you think about looking at supporting multiple lines in the event viewer? I can see others potentially wanting this as well.

[rauhs]

What do you think about looking at supporting multiple lines in the event viewer?

That would be fantastic.

The "Stack viewer" has that "F2" (ToggleNotesPaneCommand). Do you envision something like that for the event viewer? So showing some extra pane at the bottom?
That might even be nice for events with multiple (long) payloads (one payload per line instead of all on one row)

I will likely have to find a student/intern to work on this so this might take a bit to get started (I have never done any WPF)

[brianrob]

The "Stack viewer" has that "F2" (ToggleNotesPaneCommand). Do you envision something like that for the event viewer? So showing some extra pane at the bottom?
That might even be nice for events with multiple (long) payloads (one payload per line instead of all on one row)

Yeah, this sounds really nice!

I will likely have to find a student/intern to work on this so this might take a bit to get started (I have never done any WPF)

Sounds great! I haven't done any WPF either :)