[Failed at 55%: glibc-locale] Building in WSL@Win10

Platform Development
1

[Before you Start]

  1. STOP the ‘Antimalware Service Executable’ process
    This process gets too excited (CPU & Disk usage at 40~60%) because tons of files are pumping in and extracted and will slow down the whole process 2 to 3 times if not more. It won’t do anything for those linux binaries and code text in anyway, so, disable it. (gosh. Sooo quiet now…)
    Run ‘regedit’ (Windows key + R)
    Find HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender
    Create registry entry named DisableAntiSpyware, New > DWORD (32 bit) Value and set to 1.
    And, Reboot the Windows.

  2. Update WSL with the latest packages
    sudo apt-get update && sudo apt-get upgrade -y

[ERROR: Unable to parse /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb]
Alright. Please Be warned in advance for a lengthy and boring narratives for a potentially simple error with obvious solution.
I only have guess one of the following activities fixed the error, at least it seems like it for now.

After investing the error message (incl. in the orig post down below), I wanted to see how the post build process will be affected if I just comment out the line where the exception occurred.
I did:
nano +646 /home/sundew/build-webos/bitbake/lib/bb/pysh/pyshyacc.py
and commented the line:
raise sherrors.ShellSyntaxError(’’.join(msg))

And I repeated ‘bitbake webos-image’ and the build up continued without any error. Of course.

Possibility-1: Then I did ‘updatedb’ hoping locating something like ‘meta-yocto’ file to fix possible simple error in reference of: https://lists.yoctoproject.org/pipermail/yocto-builds/2012-February/012748.html
I still believe this didn’t actively fix anything but am sharing simply because I did it in between.

Possibility-2: Then I tried to simple printf for the joined msg array for quick visual check to see any abnormality and uncommented the below line, ‘#raise sherrors.ShellSyntaxError(’’.join(msg))’, and bitbake-ed again. It was supposed to halted with the same error, but it didn’t. (oing??)
Finally I deleted the printf I put, and re-bitbaked, and since then, the error condition itself seems to be long gone…

If you can shed some light, that would be wonderful for the record, so I can delete this story of a bushy kicking in the darkest night…

[ERROR on bulding gcc-source-6.2.0-6.2.0-r0: remove the strings of Windows systems’ paths in PATH env var]

I first tried to fix adding escape char (’\’) in front of all the spaces and parentheses but eventually fed up finding solution for something like ‘Files\ \(x86\)’.
(WSL. I told you this will happen some day…)
Anyway, just deleted all those strings of Windows paths and gcc-source-6.2.0-6.2.0-r0 successfully compiled.
I did (this won’t work for you 'cause of ‘/home/sundew/’):
export PATH=/home/sundew/build-webos/meta-webosose/meta-webos/scripts:/home/sundew/build-webos/oe-core/scripts:/home/sundew/build-webos/BUILD/sysroots/x86_64-linux/usr/bin/arm-webos-linux-gnueabi:/home/sundew/build-webos/BUILD/sysroots/raspberrypi3/usr/bin/crossscripts:/home/sundew/build-webos/BUILD/sysroots/x86_64-linux/usr/sbin:/home/sundew/build-webos/BUILD/sysroots/x86_64-linux/usr/bin:/home/sundew/build-webos/BUILD/sysroots/x86_64-linux/sbin:/home/sundew/build-webos/BUILD/sysroots/x86_64-linux/bin:/home/sundew/build-webos/oe-core/scripts:/home/sundew/build-webos/bitbake/bin:/home/sundew/bin:/home/sundew/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

[original post]
Have anyone tried to compile using WSL (Windows Subsystem for Linux)?
It’s Ubuntu 14.04 so I’m giving it a try.
Everything was smooth until I hit an error at ‘bitbake webos-image’:

There was single ERROR:
ERROR: Unable to parse /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb

But I confirmed that the file does exist at the specified path:
sundew@lenovo:~/build-webos$ ls -l /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb
-rw-rw-rw- 1 sundew sundew 509 Mar 20 16:06 /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb

I searched and here’s the closest I found though am not sure how to apply the proposed method ‘ignore and see what breaks’ for this codebase:
https://www.mail-archive.com/yocto@yoctoproject.org/msg38880.html

Maybe WSL falls into the same bucket as “NOT to use Linux virtual machine on Windows.” I don’t know.

Anything will help. Thanks.

[error dump]
sundew@lenovo:~/build-webos$ bitbake webos-image
Loading cache: 100% |####################################################################################| Time: 0:00:00
Loaded 17 entries from dependency cache.
WARNING: /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb: Exception during build_dependencies for do_kernel_metadata
WARNING: /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb: Error during finalise of /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb
WARNING: /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.4.bb: Exception during build_dependencies for do_kernel_metadata
WARNING: /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.4.bb: Error during finalise of /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.4.bb
ERROR: Unable to parse /home/sundew/build-webos/meta-raspberrypi/recipes-kernel/linux/linux-raspberrypi_4.9.bb
Traceback (most recent call last):
File “/home/sundew/build-webos/bitbake/lib/bb/codeparser.py”, line 341, in

[blah blah. not important…]

File “/home/sundew/build-webos/bitbake/lib/ply/yacc.py”, line 267, in LRParser.parse(input=None, lexer=<bb.pysh.pyshlex.PLYLexer object at 0x7ffa510755c0>, debug=False, tracking=0, tokenfunc=None):
else:
> return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)

File “/home/sundew/build-webos/bitbake/lib/ply/yacc.py”, line 1049, in LRParser.parseopt_notrack(input=None, lexer=<bb.pysh.pyshlex.PLYLexer object at 0x7ffa510755c0>, debug=False, tracking=0, tokenfunc=None):
errtoken.lexer = lexer
> tok = self.errorfunc(errtoken)
del errok, token, restart # Delete special functions
File “/home/sundew/build-webos/bitbake/lib/bb/pysh/pyshyacc.py”, line 646, in p_error(p=LexToken(LPARENS,’(’,0,0)):
w(’ %r\n’ % n)
> raise sherrors.ShellSyntaxError(’’.join(msg))

bb.pysh.sherrors.ShellSyntaxError: LexToken(LPARENS,’(’,0,0)
followed by:
LexToken(TOKEN,‘x86’,0,0)
LexToken(RPARENS,’)’,0,0)
LexToken(TOKEN,’/Intel/iCLS’,0,0)
LexToken(TOKEN,‘Client:/mnt/c/Program’,0,0)
LexToken(TOKEN,‘Files/Intel/iCLS’,0,0)

Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

2

Hi @sundew

I also tried to build webOS OSE in WSL@Win10 but I got the same error. webOS OSE uses openembedded build framework (http://www.openembedded.org/wiki/Main_Page). The build framework doesn’t support WSL@WIn10 yet. I’m not sure whether WSL@Win10 problem or not.

3

Now it has just started building components without error. I tried this and that and am not sure which one fixed the problem if that did, which is THE real prob…

sundew@lenovo:~/build-webos$ bitbake webos-image
Loading cache: 100% |####################################################################################| Time: 0:00:01
Loaded 2859 entries from dependency cache.
WARNING: No bb files matched BBFILE_PATTERN_meta-webos-backports-2.5 ‘^/home/sundew/build-webos/meta-webosose/meta-webos-backports/meta-webos-backports-2.5/’
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION = “1.32.0”
BUILD_SYS = “x86_64-linux”
NATIVELSBSTRING = “Ubuntu-14.04”
TARGET_SYS = “arm-webos-linux-gnueabi”
MACHINE = “raspberrypi3”
DISTRO = “webos”
DISTRO_VERSION = “1.0.g”
TUNE_FEATURES = “arm armv7ve vfp thumb neon vfpv4 callconvention-hard cortexa7”
TARGET_FPU = “hard”
WEBOS_DISTRO_RELEASE_CODENAME = “webos-master”
WEBOS_DISTRO_BUILD_ID = “unofficial”
WEBOS_DISTRO_TOPDIR_REVISION = “1e7e06b48981fbe723beb48fe3db842ac2e642cf”
WEBOS_DISTRO_TOPDIR_DESCRIBE = “”
DATETIME = “20180321044009”
meta-webos-raspberrypi = “master:7c8e550398dd3ef7ab26f59023ede177f6f9830c”
meta-raspberrypi = “morty:2a192261a914892019f4f428d7462bb3c585ebac”
meta-webos
meta-webos-backports-2.5
meta-webos-backports-2.4
meta-webos-backports-2.3 = “master:7c8e550398dd3ef7ab26f59023ede177f6f9830c”
meta-qt5 = “krogoth:f8584d7a7c90afc71484a40279aa3df651d0e04f”
meta-filesystems
meta-python
meta-networking
meta-multimedia
meta-oe = “morty:b40116cf457b88a2db14b86fda9627fb34d56ae6”
meta = “morty:1718f0a6c1de9c23660a9bebfd4420e3c4ed37e6”

WARNING: The Linux kernel on your build host was not configured to provide process I/O statistics. (CONFIG_TASK_IO_ACCOUNTING is not set)
Initialising tasks: 100% |###############################################################################| Time: 0:00:22
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
WARNING: libmpc-native-1.0.3-r0 do_fetch: Failed to fetch URL http://www.multiprecision.org/mpc/download/mpc-1.0.3.tar.gz, attempting MIRRORS if available
Currently 7 running tasks (99 of 5288) 1% |# |
0: gcc-source-6.2.0-6.2.0-r0 do_unpack - 491s (pid 3805)
1: binutils-cross-arm-2.27-r0 do_fetch (pid 3844) 77% |############################################### |
2: glibc-initial-2.24-r0 do_fetch (pid 3981) 95% |############################################################### |
3: glibc-2.24-r0 do_fetch - 405s (pid 4031)
4: gettext-0.19.8.1-r0 do_unpack - 151s (pid 4869)
5: ncurses-native-6.0+20160625-r0 do_unpack - 0s (pid 5152)
6: quilt-native-0.64-r0 do_populate_sysroot - 0s (pid 5155)

4

Great :slight_smile: Could you let me know when you finish building on WSL@win10?

5

Of course, I will. It’s now 8%, and I had a couple more hiccups I overcame I can share how.
Maybe we can make this thread as a guide helping ppl using WSL as building env if succeed.

6

I can recognized few Korean names in this forum. 안녕하세요?
If you have Naver (a Korean portal) membership, you can join us in PIPC, a SBC community@:

http://cafe.naver.com/pipc/22528

1 Like
7

I’m having it fail almost immediately in codeparser.py, line 341, as was said in the original message.

I’ve attempted other OE builds in WSL (LuneOS, specifically, the old Open webOS also), and have run into various other problems, but they did not fail immediately.

Other people have attempted to use WSL to build other OE systems, and for the current release of WSL, it’s just not likely to work at all unless you’re building something very very small. The NTFS system that WSL works on top of, prevents open files from being moved, which is not a problem in real Linux. Because that’s not a problem in real Linux, there’s a lot of software involved that doesn’t expect that to be a problem.

That said, if we can get the base process working to any reasonable degree, then it’s possible to fix the other broken pieces. I know Microsoft expects to have some of the file locking problems wrt NTFS solved over the next few major releases, but they are on a 6 month release schedule, and it may not be possible to ever get it 100% fixed at the WSL level – so some changes may need to be done in other software to make them work with the NTFS file locking limitations.

8

That’s interesting. Thanks for sharing.
Well, I guess than only time will tell. It’s now 53% completed. I’ve found it stopped 4 times so far, mostly with fetch errors, and usually the problem disappeared when I restarted it.
We will see.

9

NTFS file locking is going to be a problem. Even relatively simple things like ‘npm’ have problems with it in WSL. That most software works in WSL, shows that it is solvable, though.

If you do figure out what you did to get the build to start, it’d be great to share. I’d be happy to get some more info up on the WSL bug tracker, and maybe we can get some of the problems sorted out. I know it’s improved a LOT in the last two Windows releases, but there’s still plenty of improvement to be done.

10

I am stuck with mainly these 3 now. Will try to deal with one by one with module bitbaking.
Where can I find the component-name for each? I tried ‘bitbake qt/qt5/qtbase/qtbase-git/qtbase_git’ but didn’t work and all returned: ERROR: Nothing PROVIDES ‘qt*****’

NOTE: Tasks Summary: Attempted 2174 tasks of which 2157 didn’t need to be rerun and 3 failed.
Summary: 3 tasks failed:
/home/sundew/build-webos/meta-qt5/recipes-qt/qt5/qtbase_git.bb:do_compile
/home/sundew/build-webos/meta-webosose/meta-webos/recipes-webos/chromium/chromium53.bb:do_compile
/home/sundew/build-webos/oe-core/meta/recipes-core/glibc/glibc-locale_2.24.bb:do_package

11

It should be qtbase, chromium53 and glibc-locale. You don’t need to provide the whole path. It you ran the script to setup the environment BitBake will know where to find the recipe within the layers. Learning curve with BitBake can be a bit steep, but it’s a very nice & versatile way to build things!

12

Thanks a lot. Yes. It seems very powerful.
I learned many (not all) components are under ‘~/build-webos/BUILD/work/raspberrypi3-webos-linux-gnueabi’ and the directory name is component name.

13

OK. Here’s an update. About the 3 failures I mentioned in the previous post, I found qtbase and chromium failed because of the glibc-locale failure and am still struggling at finding why. Here’s the log if you are interested:

sundew@lenovo:~/build-webos$ bitbake glibc-locale
…
Initialising tasks: 100% |#####################################################################################################| Time: 0:00:04
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: glibc-locale-2.24-r0 do_package: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: ‘exec_python_func() autogenerated’, lineno: 2, function:
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: ‘/home/sundew/build-webos/oe-core/meta/classes/sstate.bbclass’, lineno: 676, function: sstate_task_postfunc
0672: bb.build.exec_func(intercept, d, (d.getVar(“WORKDIR”, True),))
0673: omask = os.umask(0o002)
0674: if omask != 0o002:
0675: bb.note(“Using umask 0o002 (not %0o) for sstate packaging” % omask)
*** 0676: sstate_package(shared_state, d)
0677: os.umask(omask)
0678:}
0679:sstate_task_postfunc[dirs] = “${WORKDIR}”
0680:
File: ‘/home/sundew/build-webos/oe-core/meta/classes/sstate.bbclass’, lineno: 599, function: sstate_package
0595: for plain in ss[‘plaindirs’]:
0596: pdir = plain.replace(workdir, sstatebuild)
0597: bb.utils.mkdirhier(plain)
0598: bb.utils.mkdirhier(pdir)
*** 0599: oe.path.copyhardlinktree(plain, pdir)
0600:
0601: d.setVar(‘SSTATE_BUILDDIR’, sstatebuild)
0602: d.setVar(‘SSTATE_PKG’, sstatepkg)
0603:
File: ‘/home/sundew/build-webos/oe-core/meta/lib/oe/path.py’, lineno: 93, function: copyhardlinktree
0089: else:
0090: source = src
0091: s_dir = os.getcwd()
0092: cmd = ‘cp -afl --preserve=xattr %s %s’ % (source, os.path.realpath(dst))
*** 0093: subprocess.check_output(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT)
0094: else:
0095: copytree(src, dst)
0096:
0097:def remove(path, recurse=True):
File: ‘/usr/lib/python3.4/subprocess.py’, lineno: 620, function: check_output
0616: process.wait()
0617: raise
0618: retcode = process.poll()
0619: if retcode:
*** 0620: raise CalledProcessError(retcode, process.args, output=output)
0621: return output
0622:
0623:
0624:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command ‘cp -afl --preserve=xattr ./* /home/sundew/build-webos/BUILD/work/cortexa7hf-neon-vfpv4-webos-linux-gnueabi/glibc-locale/2.24-r0/sstate-build-package/packages-split’ returned non-zero exit status 1

…

ERROR: glibc-locale-2.24-r0 do_package: Function failed: sstate_task_postfunc
ERROR: Logfile of failure stored in: /home/sundew/build-webos/BUILD/work/cortexa7hf-neon-vfpv4-webos-linux-gnueabi/glibc-locale/2.24-r0/temp/log.do_package.7528
ERROR: Task (/home/sundew/build-webos/oe-core/meta/recipes-core/glibc/glibc-locale_2.24.bb:do_package) failed with exit code ‘1’

NOTE: Tasks Summary: Attempted 397 tasks of which 396 didn’t need to be rerun and 1 failed.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 29.742190837860107 seconds
NOTE: Build completion summary:
NOTE: do_package: 0.0% sstate reuse(0 setscene, 1 scratch)

Summary: 1 task failed:
/home/sundew/build-webos/oe-core/meta/recipes-core/glibc/glibc-locale_2.24.bb:do_package
Summary: There were 2 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

14

links can be a little wonky in WSL, again, due to how NTFS handles things differently versus a real ext4. It would be good if you can get the output from ‘strace’ on the bitbake run, and try to narrow it down to what syscall is failing that shouldn’t be… the error itself doesn’t seem to be too intuitive. :frowning:

15

Yes I agree. I got the strace output. Not sure how much more I’ll swim in it though…
Before I leave from it, I really wanted to help you and others to overcome the first hurdle, so I attempted adding a new user and fresh cloned the git repo and reconfigured to mimic the virgin state once I had with the error, however, no, with the bitbake command, it went straight to the initializing mode without any error, and it seems really disappeared no matter what that was, and I no longer have opportunity for a proper documentation. sorry.

16

That’s interesting. I wonder if deleting and reinstalling my Ubuntu WSL would help. There might be more successes if using an Insider Windows build too, there is a lot of ongoing work in WSL, although none of the bugs that I’m subscribed to on github regarding file locking have really had any fixes that have been mentioned, recently. WSL is fantastic for what it does, but it’s definitely not 100% :slight_smile:

A more likely method to get up and running in Windows quickly, would be to use a Ubuntu VM, Docker, and/or Vagrant. Although Docker is quite popular these days, I think either a full standard VM or Vagrant is better suited for doing development, whereas Docker is more suited for software deployments.

That said, i’d really love to see if it’s possible over the next few releases of WSL, WebOS, OpenEmbedded, etc … to get it to work in WSL. But for the time being, I think Vagrant, VMs, or even Docker are better choices to actually get work done in.

17

FYI

Let’s keep track of https://github.com/Microsoft/WSL/issues/3051

1 Like