Yes, I agree that there is little harm in gathering too much code. I have tried out just scanning data memory for values that refer to addresses within the region marked as code and disassembling from those points, as well as scanning the instructions I traverse for any immediate values in the same range.
Looking through Wikipedia at least, it's not exactly clear to me. They have separate pages for 'binary recompiler' and 'binary translation' that link to each other, and the latter is more about going between architectures (which is the main objective here).
[post author] I went down some similar paths in retrowin32, though 32-bit x86 is likely easier.
I was also surprised by how much goop there is between startup and main. In retrowin32 I just implemented it all, though I wonder how much I could get away with not running it in the Theseus replace-some-parts model.
I mostly relied on my own x86 emulator, but I also implemented the thunking between 64-bit and 32-bit mode just to see how it was. It definitely was some asm but once I wrapped my head around it it wasn't so bad, check out the 'trans64' and 'trans32' snippets in https://github.com/evmar/retrowin32/blob/ffd8665795ae6c6bdd7... for I believe all of it. One reframing that helped me (after a few false starts) was to put as much code as possible in my high-level language and just use asm to bridge to it.
Yeah, 32-bit x86 is somewhat easier because everything's in the same flat address space, and you at least have a system-wide code32 gdt entry that means you can ignore futzing around with the ldt. 16-bit means you get to deal with segmented memory, and the cherry on top is that gdb just stops being useful since it doesn't know anything about segmented memory (I don't think Linux even makes it possible to query the LDT of another process, even with ptrace, to be fair).
As for trying to ignore before main... well, the main benefit for me was being able to avoid emulating DOS interrupts entirely, between skipping the calls to set up various global variables, stubbing out some of the libc implementations, and manually marking in the emulator that code page X was 32-bit (something else that sends tools in a tizzy, a function switching from 16-bit to 32-bit mid-assembly code).
16-bit is weird and kinda fun to work with at times... but there's also a reason that progress on this is incredibly slow for me.
Slow progress is fine, it took me like two years to get where I got! (Not that I was working on it full time or anything, but also there were just many false starts and I had no idea what I was doing...)
It depends on what results you’re expecting! Relative to an interpreter, even the simplest unoptimized translation of that code is already significantly more efficient code at runtime.
In the post there is a godbolt link showing a compiler inlining a simple add, but a real implementation of x86 add would be much more complex.
I have read other projects where the authors put some effort into getting exactly the machine code they wanted out. For example, maybe you want the virtual regs.eax to actually exist in a machine register, and one way you might be able to convince a compiler to do that is by passing it around as a function parameter everywhere instead as a struct element. I have not investigated this myself.
I’m not too familiar with Firefox builds. Why are clobber builds common? At first glance it seems weird to add a cache around your build system vs fixing your build system.
Though I'm not actively working with Firefox so can't speak for their use cases, one important use case for clobber builds is CI.
I'm the author of BuildCache, and where I work we make thousands of clobber builds every day in our CI. Caching helps tremendously for keeping build times short.
There are a few use cases for local development too. For instance if you switch between git branches you may have to make near full rebuilds (e.g. in C++ if some header file is touched that gets included by many files).
Another advantage as a local dev is that you can tap into the central CI cache and when you pull the latest trunk and build it, chances are that the CI system has already built that version (e.g. as part of a merge gate) so you will get cache hits.
I see, I might be confused by the terminology. "clobber" to me suggests intentionally trying to throw away cached results (clobbering what you have), but it sounds like you might just use it to mean builds where you don't have any existing build state already present.
Sorry for being unclear. I'm using Firefox build system lingo without explanations. It's from the command `./mach clobber`, which is similar but not the same as `make clean`. I use 'clobber build' as "a build with no existing build state" and the qualifiers "cold" and "warm" to indicate if cache is empty or filled.
I'm afraid I still don't understand. One factor is having fewer features and not looking for obsolete files, that I can understand. I guess the other thing is using better rules to figure out when a file truly need to be rebuilt?
To be honest, it's not clear to me why other systems are not faster. Ninja is relatively straightforward but also not too clever.
Now that I think about it, I did write more about some of the performance stuff we did here: https://aosabook.org/en/posa/ninja.html Looking back over that, I guess we did do some lower-level optimization work. I think a lot of it was just coming at it from a performance mindset.
I might be missing your sarcasm, but this is a common approach for large scale builds. Virtual filesystems are used to provide a pre-computed tree hash as a xattr. In a more typical case, you can read the git tree hash.
Not sure it was meant as sarcasm really. I just think so many build (and other) problems could have been avoided it a file hash was available on every file by default.
In the current POSIX paradigm yes, it would be expensive. But if the hash was defined as the hash of fixed blocks, it wouldn't be expensive. The raciness depends, a lot, on the semantics we would define. (In the context of a build system, it's no different than that the file could get a new mtime after we read the mtime.)
I had a similar experience implementing simd instructions in my emulator, where I needed to break apart a 64-bit value into four eight-bit values, do an operation on each value, then pack it back together. My first implementation did it with all the bit shifts you’d expect, but my second one used two helpers to unpack into an array, map on the array to a second array, and pack the array again. The optimized output was basically the same.
I don't know the technical details, but the kernel docs say "It exists because implementation in user-space, using existing tools, cannot match Windows performance while offering accurate semantics."
https://docs.kernel.org/userspace-api/ntsync.html
reply