https://codereview.chromium.org/1784243006/ added the ALLOW_TIMERS define, and I forgot to add it to Makefile and CMakeLists.txt.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1788873004 .

Several things are done here:

1. Move timer support to be guarded by the ALLOW_TIMERS define, or the BuildDefs::timers() constexpr method.

2. Add a NODUMP build configuration to control whether dump support is built in.  So "make -f Makefile.standalone NODUMP=1 NOASSERT=1" is pretty close to a MINIMAL build with timer support.

3. Add some missing timers: alloca analysis, RMW analysis, helper call pre-lowering, load optimization analysis.  These omitted pass timings were being rolled up into the "O2" bucket.

4. Add timers around push and pop operations on the translate queue and the emit queue.

5. Refactor the clumsy code to push/pop function timers (as opposed to pass timers), so that it fits into the nice RAII TimerMarker class like the pass timers.

6. It turns out that even with MINIMAL or NODUMP builds, we still construct a longish std::string every time Cfg::dump() is called, even though the string isn't used in MINIMAL/NODUMP mode.  The dump() arg might as well be a const char * arg instead.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4360
R=kschimpf@google.com

Review URL: https://codereview.chromium.org/1784243006 .

In some cases, Subzero needs to insert into a std::vector at a particular index, resizing the vector as necessary.  It appears that our vector implementation sets the capacity to exactly the size when growing the vector, without leaving any extra capacity.  This causes lots of mallocs and recopies each time the vector size is increased.  (Adding elements via push_back() or emplace_back() doesn't seem to have that behavior.)

We help this by reserving some extra space before resizing - bump to the next power of 2 up to some point, then bump to the next multiple of a chunk size beyond that point.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4360
R=kschimpf@google.com

Review URL: https://codereview.chromium.org/1783113002 .

A fresh checkout of native_client lacks some components that Subzero's "make -f Makefile.standalone presubmit" needs.

Add explicit checks for these components, and when missing, print suggestions for how to create them.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4359
R=jpp@chromium.org, smklein@chromium.org

Review URL: https://codereview.chromium.org/1782343003 .

This CL updates "isPNaClABIExternalName" -- Subzero checked for the symbol
"__pnacl_pso_root" as a function, but it is a declaration, and should be
checked as one. Additionally, when the PNaClTranslator is verifying the linkage
of declarations, allow "__pnacl_pso_root" to be flipped to external as a special
case.

Previously, translating a pso with --use-sz caused the warning:
"cannot find entry symbol '__pnacl_pso_root'".
That warning is removed with this CL.

Fixes revert from https://codereview.chromium.org/1745783002/

TEST=external_declaration.ll
BUG=https://bugs.chromium.org/p/nativeclient/issues/detail?id=4351
R=kschimpf@google.com, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1774383002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4360
BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4077
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1783893002 .

Patch from John Porto <jpp@chromium.org>.

This reverts commit 5526c171 (https://codereview.chromium.org/1778663003) and implements it per jpp's suggestion.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1780773003 .

Additionally, refactor "GetObjdumpCmd" and "GetObjcopyCmd".

BUG=https://bugs.chromium.org/p/nativeclient/issues/detail?id=4361
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1777103002 .

1. Subzero constructs many strings based in part on function name.  When function names are not present (as in properly finalized pexes), they are synthesized as something like "Function12345".  We can shorten these strings to e.g. "F12345" by using --default-function-prefix=F .  Similar for global variable names.

Using short strings makes it much less likely to have to use malloc.  As such, we force that to be the default in the browser translator build.  For perf-testing the command-line version, the user can just add the option manually for now.

Ultimately, we should avoid use of strings in this way.

2. The register allocator uses a few instances of llvm::SmallVector that are sized too small and therefore end up using malloc.  This can be fixed in a clean way, and there is a TODO for it, but in the meantime we just bump the size.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4360
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1776343004 .

le32-nacl-objdump has been deprecated, and should no longer be used.
Instead, "arm-nacl-objdump" is being used. "le32-nacl-objdump" used to be
a hard link to "arm-nacl-objdump", but has since been deleted in NaCl's
"toolchain_build_pnacl.py" script.

R=phosek@chromium.org, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1776843002 .

BUG= none
TBR=jpp@chromium.org

Review URL: https://codereview.chromium.org/1778663003 .

The ConstantRelocatable objects for pushing local labels are allocated from the Assembler arena, and are no longer pooled, which restricts the memory growth from sandboxing x86-64 calls.

Because the Assembler arena is destroyed while the fixups are still active, these fixups have to be fixed up by holding a pointer to the symbol rather than the constant.

On the 10MB test pexe, the overall growth by the end is ~20MB, instead of ~130MB as before.

This also partially fixes an existing bug with arm32/nonsfi/iasm, exposed by running cross tests and forcing iasm output.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1773503003 .

The previous implementation was charging about 24% more time that it
should to the function parser. The cause was that the time to "queue" the
parsed functions, and the time to emit the assembled code (again
including "queue" time) was not accounted for. About 15% was going to
queuing costs, and 7% to emitting the ELF file.

This CL adds timing of function translateFunctions, which captures most
of the queueing costs, and timing for each of the major ELF emission
functions (emitELF). This allows the corresponding costs to be better
bucketed, and not charged to the time it takes to parse functions in
bitcode files.

Bug=None

R=jpp@chromium.org, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1775603002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=eholk@chromium.org, kschimpf@google.com, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1768823002 .

For the Om1 spec tests, we were generating commands like this:

  ./pydir/szbuild_spec2k.py -v -Om1 --target=x8632 -O2 --filetype=obj 253.perlbmk

where -Om1 and -O2 both appear.  Python's argparse allows an argument to be set multiple times, and the last one would win, i.e. -O2 is used instead of -Om1.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1766603002 .

This cleans up the whole --prefix handling mechanism, so that all mangling is done based on logic in the parser instead of scattered all over the code base.  Another nice side effect is that it allows some getName() style functions to return a const string reference instead of a string copy.

Also, moves ClFlags into a static field of GlobalContext, i.e. makes it global, so that these constant flags can be accessed without having to plumb a GlobalContext object.  Note that some of the ClFlags and ClFlagsExtra plumbing in the compile server classes could be simplified to directly use the corresponding static fields, but this is left mostly as is for now, for when we do a proper separation between core and supplemental flags.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1766233002 .

With "-verbose cpool", at the end it dumps counts of the number of times each pooled constant was looked up.  The output is grouped by type and sorted by constant value, but it's easy to run it through "sort -nr" to see it ordered by popularity.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1768173002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1755333002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1747243003 .

BUG= none
R=eholk@chromium.org, jpp@chromium.org

Review URL: https://codereview.chromium.org/1747333002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1745393002 .

There were a lot of unnecessary copying and resizing and sloppiness in the use of BitVector for liveness analysis.  This tries to reduce the amount of copying and associated memory allocation.

Also, works around a ConstantRelocatable hashing problem that was causing a huge slowdown in MINIMAL mode for the vector_shuffle scons test.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1746613002 .

BUG= None
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1741733003 .

The problem is that when switch lowering decides to use a JumpTable and the switch variable is i64 on x86-64, the lowering tries to movzx the i64 variable into an i32 variable, and the Movzx ctor asserts.

This happens when translating pnacl-llc.pexe, but luckily it is also triggered in the existing adv-switch.ll test.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1743133002 .

One is rarely interested in seeing dump output regarding global initializers, so we add an additional verbose flag, and don't include it in the "-verbose most" set.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1744913002 .

Along with the nexe equivalents of pnacl-sz, also build the browser-integrated versions that can be dropped in as replacements.  Note that these are not exact replacements, as they likely support all targets.

In "make presubmit", instead build sandboxed translators in DEBUG=1 mode, so they can build in parallel with the assembler unit tests.

Add a "make bloat-sb" target that does the "make bloat" equivalent for the sandboxed translators.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1746593002 .

This reverts commit 352db935 from https://codereview.chromium.org/1740033002/ .

That commit broke "make -f Makefile.standalone check-lit check-xtest".

BUG= none
R=smklein@chromium.org

Review URL: https://codereview.chromium.org/1745783002 .

Adds "-verbose mem" to enable printing the amount of CfgLocalAllocator memory allocated, each time Cfg::dump() is called.

"-verbose mem,status" is a good way to get summary info.

BUG= none
R=kschimpf@google.com

Review URL: https://codereview.chromium.org/1743043002 .

This CL updates "isPNaClABIExternalName" -- Subzero checked for the symbol
"__pnacl_pso_root" as a function, but it is a declaration, and should be
checked as one. Additionally, when the PNaClTranslator is verifying the linkage
of declarations, allow "__pnacl_pso_root" to be flipped to external as a special
case.

TEST=pnacl-translate -pso --use-sz -arch x86-32-nonsfi test_pll.final.pso
BUG=https://bugs.chromium.org/p/nativeclient/issues/detail?id=4351
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1740033002 .

BUG=

Review URL: https://codereview.chromium.org/1742833002 .

1) Clones llvm::BitVector, and makes it Allocator aware (using the
CfgLocalAllocator<>)

2) Uses mallopt to set the malloc granularity. The default granularity
is too small, which forces too many mmap calls.

BUG=
R=sehr@chromium.org, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1738683003 .

See failed build at https://build.chromium.org/p/tryserver.nacl/builders/nacl-toolchain-win7-pnacl-x86_64/builds/3441 .

BUG= none
TBR=jpp@chromium.org

Review URL: https://codereview.chromium.org/1732233002 .

Instead of "./pnacl-sz <args>", one can run:

  ../../../run.py ./pnacl-sz.x8632.nexe <args>
or
  ../../../run.py ./pnacl-sz.x8664.nexe <args>

Hopefully the translator performance characteristics are close to that of the browser hookup.

Adds sb builds to two of the presubmit configurations.

BUG= none
R=jpp@chromium.org

Review URL: https://codereview.chromium.org/1738633002 .

Introduces Ice::SmallBitVector.

Modifies ConstantRelocatables so that known offsets (i.e.,
not offsets to the code stream) do no require GlobalContext
allocations.

Modifies Cfg-local containers to use the CfgLocalAllocator.

BUG=
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1738443002 .

TBR=jpp@chromium.org

BUG= none

Review URL: https://codereview.chromium.org/1730263002 .

This is part of ARM patch:
https://codereview.chromium.org/1151663004/

BUG=
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1724643002 .

Patch from Reed Kotler <rkotlerimgtec@gmail.com>.

Implement 64 bit multiply in mips32 and, in addition, add the lo/hi registers which are also used for other 64 bit math such as div, rem.

BUG=
R=jpp@chromium.org, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1716483003 .

Patch from Reed Kotler <rkotlerimgtec@gmail.com>.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=eholk@chromium.org, stichnot@chromium.org

Review URL: https://codereview.chromium.org/1708753002 .

This removes most of the #ifndef ARM32 directives so we get more
thorough testing. It still uses fewer iterations due to running on
Qemu, but I did manually check to be sure that the tests pass without
any special casing at all.

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=jpp@chromium.org, kschimpf@google.com

Review URL: https://codereview.chromium.org/1708903002 .

BUG= https://bugs.chromium.org/p/nativeclient/issues/detail?id=4076
R=stichnot@chromium.org

Review URL: https://codereview.chromium.org/1685253003 .