Bitbake is a clever idea not taken far enough. The caching is broken, and bit me multiple times, and the bash-but-not-just-bash configuration language is horrible and yes, the abstraction levels are convoluted and disorienting -- there's no clear way to find out where all these settings are coming in from. By the time bitbake is taken far enough it becomes Nix, and at that point they should have just used Nix to begin with and saved a decade of effort.
A start at doing something like yoctolinux but Right is @telent
's NixWRT(0)(1), built using Nix and a lightly patched nixpkgs.
It does less than bitbake so far, but without even trying it out, I know that its caching is not broken, because it is using Nix.