First, let's acknowledge that the plural of "Cracker Jack" is probably "Cracker Jack" That's typical with brand names.
But people actually seem to use a pluralized form of the name - so often, at least, that I only realized the brand name was "Cracker Jack" and not "Cracker Jacks" when fact-checking this post.
So for the sake of argument, let's assume the word needs a plural.
"Crackers Jack" is similar in construction to "Attorneys General," so that seems like a good place to start. Once we understand the construction of "Attorneys General" we can see if the same logic apples to "Crackers Jack."
"Attorney General" is a compound noun, and "General" acts as an adjective modifying "Attorney." (What kind of attorney? the general one.) Since you pluralize nouns, not adjectives, the plural form is "Attorneys General."
Here’s another example: "Able seaman." This is also a compound noun. "Able" is an adjective modifying "seaman." Thus the plural form is "Able seamen," not "Ables seaman."
The rule then seems to be that you capitalize the head, not just the first word.
Based on this, you could argue that "Cracker" is being used as an adjective (what kind of Jack? A cracker one, which has a … different connotation nowadays) and thus "Cracker Jacks" — precisely they way everyone says it anyway.
Strictly speaking, I think the real answer to this is that the plural of "Cracker Jack" is the same as the singular, and any other plural form is never grammatically correct. But now, at least, maybe "Attorneys General" doesn't seem so strange.