Repository navigation
is the test suite something that could be factored out/generalized? #40
Description
You've done some incredible stuff in there making it possible to generate dumps, use dump_syms, and then load dumps again. I've been wanting a tool that is two binaries, minidump-harness and futility.
futility
(it seems like src/bin/test.rs is basically this)
The futile utility application, providing a "crash suite"
- takes a simple cli arg choosing which func to call that will die in an interesting way
- we can build for whatever platform
- run dump_syms on
minidump-harness
(it seems like tests/ is kinda this)
Orchestrates building/analyzing/torturing futility and storing/diffing the results. it would serve three roles:
- regression tests (running in ci)
- getting a particular intermediate result of the pipeline for inspection
- enabling us to quickly test/diff a change across our entire pipeline
The first is straight-forward and you mostly have that setup (although #30 still needs to be done). The second is more subtle. You can see my initial attempts at this problem with my socc-pair utility which largely automates diffing/benching a local version of minidump-stackwalk against what mozilla has in production. This is great for testing if I fixed a bug in production, but it's still more manual than I'd like.
The Dream
$ minidump-harness run base-trial
<spews a bunch of shit into a dir called base-trial>
<now i make some changes like [patch]ing local changes to rust-minidump/symbolic/dump_syms...>
$ minidump-harness run patched-trial
<spews a bunch of shit into a dir called patched-trial>
$ minidump-harness diff base-trial patched-trial
<spits out a report of what changed>
The Design
minidump-harness run $DIR would:
- build futility, putting the target in $DIR/target
- run dump-syms on it, putting the symbols in $DIR/symbols
- run every crash suite it has, putting the dumps in $DIR/dumps/$suite.dmp
- run
minidump-stackwalk --cyborg --verbose=trace --symbols-path=$DIR/symbolson each minidump, putting:- json in $DIR/results/$suite.json
- human in $DIR/results/$suite.txt
- logs in $DIR/results/$suite.log.txt
minidump-harness diff $DIR1 $DIR2 would:
- spit out a diff of particular details (default would probably be what socc-pair does?)
- but ideally would streamline other diff/filter queries like "did the human output change" or "just tell me about this suite"
- probably also want to be able to cargo-insta (snapshot) some of these results?
Right now you have some of these things, but a lot of the test implementations are in the harness and kind of manually adhoced. I don't fully understand your design, and all the constraints there are. I'm happy to do a lot of legwork here, but you've clearly done a lot stuff that's worth using/extending instead of me kicking off a green-field project.
Activity
Gankra commentedon Jul 7, 2022
cc @mstange and @Jake-Shadle
Jake-Shadle commentedon Jul 7, 2022
I think something like this would be great. The MacOS dump_syms stuff I added was loosely based on the end-to-end testing I did in https://github.com/EmbarkStudios/crash-handling/tree/main/minidumper-test though there I at the moment just doing the basic "does this minidump contain the expected crash reason" without checking the symbolicated stack traces for the thread(s) also match, but that was something I wanted to add eventually.
I guess the one big difference to me is that the dream scenario you specified above feels a bit too manual for me, but maybe this is just a different workflow for doing changes before making automated tests?
Gankra commentedon Jul 7, 2022
yeah i made the dream slightly manual just so we could do absolutely whatever we want and extract only the parts we care about for individual runs.
i.e. "this old dump_syms version had this problem, I want a sym file showing that for my tests, so I will
[patch]symbolic back to that version and dominidump-harness runand grab thesymfile it spits out for my test-suite"making automated tests I think would look closer to
cargo instasnapshots, where we have a copy of the outputs we care about saved in one dir, and the tests justminidump-harness runand diff against the snapshot.Gankra commentedon Jul 7, 2022
basically it would be useful as a
Gankra commentedon Jul 14, 2022
Ok here's a rough diagram of what a full deployment architecture looks like.
I've broken the architecture into 3 major sections:
The client and server are further broken up into separate processes.
Rough atlas:
Note that Mozilla (and Sentry?) don't use any of the "client" stuff right now except for minidump-writer in some specific situations. Migrating to minidump-writer completely is desirable for Mozilla (kill breakpad), but the other stuff is less clearly important to us.
graph TD; subgraph build cargo dump_syms http-symbol-sender(http-symbol-sender) binary{.exe/.pdb} sym{.sym} end subgraph client subgraph app YOUR_ACTUAL_APP(YOUR ACTUAL APP) minidumper::Client crash-handler end subgraph crash-monitor minidumper::Service minidump-writer http-dump-sender(http-dump-sender) context{CrashContext} dump{.dump} end end subgraph server subgraph crash-server http-dump-receiver(http-dump-receiver) minidump-processor minidump structMinidump{Minidump} report{report.json} end subgraph symbol-server tecken(tecken) http-symbol-receiver(http-symbol-receiver) end subgraph crash-stats socorro(socorro) end end cargo -.-> binary binary -.-> dump_syms binary -.-> deploy(deploy) dump_syms -.-> sym sym -.-> http-symbol-sender crash-handler ---> minidumper::Client minidumper::Client ---> minidumper::Service minidumper::Service ---> minidump-writer minidump-writer -.-> dump dump -.-> http-dump-sender minidumper::Service ---> http-dump-sender minidumper::Service -.-> context context -.-> minidump-writer http-dump-receiver ---> minidump-processor http-dump-receiver ---> minidump minidump -.-> structMinidump structMinidump -.-> minidump-processor tecken -. .sym .-> minidump-processor minidump-processor -.-> report report -.-> http-dump-receiver http-dump-receiver -. report.json .-> socorro http-symbol-sender -. .sym .-> http-symbol-receiver http-dump-sender -. .dump .-> http-dump-receivergraph source