Added Cyg-Win

This commit is contained in:
Frank Harris 2026-06-06 18:46:40 -04:00
parent 82cbc206eb
commit 413c315806
10586 changed files with 3806249 additions and 0 deletions

View file

@ -0,0 +1,58 @@
Authors of XZ Utils
===================
XZ Utils is developed and maintained by
Lasse Collin <lasse.collin@tukaani.org>.
Major parts of liblzma are based on code written by Igor Pavlov,
specifically the LZMA SDK <https://7-zip.org/sdk.html>. Without
this code, XZ Utils wouldn't exist.
The SHA-256 implementation in liblzma is based on code written by
Wei Dai in Crypto++ Library <https://www.cryptopp.com/>.
A few scripts have been adapted from GNU gzip. The original
versions were written by Jean-loup Gailly, Charles Levert, and
Paul Eggert. Andrew Dudman helped adapting the scripts and their
man pages for XZ Utils.
The initial version of the threaded .xz decompressor was written
by Sebastian Andrzej Siewior.
The initial version of the .lz (lzip) decoder was written
by Michał Górny.
Architecture-specific CRC optimizations were contributed by
Ilya Kurdyukov, Chenxi Mao, and Xi Ruoyao.
Other authors:
- Jonathan Nieder
- Joachim Henke
Special author: Jia Tan was a co-maintainer in 2022-2024. He and
the team behind him inserted a backdoor (CVE-2024-3094) into
XZ Utils 5.6.0 and 5.6.1 releases. He suddenly disappeared when
this was discovered.
Many people have contributed improvements or reported bugs.
Most of these people are mentioned in the file THANKS.
The translations of the command line tools and man pages have been
contributed by many people via the Translation Project:
- https://translationproject.org/domain/xz.html
- https://translationproject.org/domain/xz-man.html
The authors of the translated man pages are in the header comments
of the man page files. In the source package, the authors of the
translations are in po/*.po and po4a/*.po files.
Third-party code whose authors aren't listed here:
- GNU getopt_long() in the 'lib' directory is included for
platforms that don't have a usable getopt_long().
- The build system files from GNU Autoconf, GNU Automake,
GNU Libtool, GNU Gettext, Autoconf Archive, and related files.

View file

@ -0,0 +1,70 @@
XZ Utils Licensing
==================
Different licenses apply to different files in this package. Here
is a summary of which licenses apply to which parts of this package:
- liblzma is under the BSD Zero Clause License (0BSD).
- The command line tools xz, xzdec, lzmadec, and lzmainfo are
under 0BSD except that, on systems that don't have a usable
getopt_long, GNU getopt_long is compiled and linked in from the
'lib' directory. The getopt_long code is under GNU LGPLv2.1+.
- The scripts to grep, diff, and view compressed files have been
adapted from GNU gzip. These scripts (xzgrep, xzdiff, xzless,
and xzmore) are under GNU GPLv2+. The man pages of the scripts
are under 0BSD; they aren't based on the man pages of GNU gzip.
- Most of the XZ Utils specific documentation that is in
plain text files (like README, INSTALL, PACKAGERS, NEWS,
and ChangeLog) are under 0BSD unless stated otherwise in
the file itself. The files xz-file-format.txt and
lzma-file-format.xt are in the public domain but may
be distributed under the terms of 0BSD too.
- Translated messages and man pages are under 0BSD except that
some old translations are in the public domain.
- Test files and test code in the 'tests' directory, and
debugging utilities in the 'debug' directory are under
the BSD Zero Clause License (0BSD).
- The GNU Autotools based build system contains files that are
under GNU GPLv2+, GNU GPLv3+, and a few permissive licenses.
These files don't affect the licensing of the binaries being
built.
- The 'extra' directory contains files that are under various
free software licenses. These aren't built or installed as
part of XZ Utils.
The following command may be helpful in finding per-file license
information. It works on xz.git and on a clean file tree extracted
from a release tarball.
sh build-aux/license-check.sh -v
For the files under the BSD Zero Clause License (0BSD), if
a copyright notice is needed, the following is sufficient:
Copyright (C) The XZ Utils authors and contributors
If you copy significant amounts of 0BSD-licensed code from XZ Utils
into your project, acknowledging this somewhere in your software is
polite (especially if it is proprietary, non-free software), but
it is not legally required by the license terms. Here is an example
of a good notice to put into "about box" or into documentation:
This software includes code from XZ Utils <https://tukaani.org/xz/>.
The following license texts are included in the following files:
- COPYING.0BSD: BSD Zero Clause License
- COPYING.LGPLv2.1: GNU Lesser General Public License version 2.1
- COPYING.GPLv2: GNU General Public License version 2
- COPYING.GPLv3: GNU General Public License version 3
If you have questions, don't hesitate to ask for more information.
The contact information is in the README file.

View file

@ -0,0 +1,11 @@
Permission to use, copy, modify, and/or distribute this
software for any purpose with or without fee is hereby granted.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

View file

@ -0,0 +1,338 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
<https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Moe Ghoul>, 1 April 1989
Moe Ghoul, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.

View file

@ -0,0 +1,501 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 2.1, February 1999
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
<https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
[This is the first released version of the Lesser GPL. It also counts
as the successor of the GNU Library Public License, version 2, hence
the version number 2.1.]
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
Licenses are intended to guarantee your freedom to share and change
free software--to make sure the software is free for all its users.
This license, the Lesser General Public License, applies to some
specially designated software packages--typically libraries--of the
Free Software Foundation and other authors who decide to use it. You
can use it too, but we suggest you first think carefully about whether
this license or the ordinary General Public License is the better
strategy to use in any particular case, based on the explanations below.
When we speak of free software, we are referring to freedom of use,
not price. Our General Public Licenses are designed to make sure that
you have the freedom to distribute copies of free software (and charge
for this service if you wish); that you receive source code or can get
it if you want it; that you can change the software and use pieces of
it in new free programs; and that you are informed that you can do
these things.
To protect your rights, we need to make restrictions that forbid
distributors to deny you these rights or to ask you to surrender these
rights. These restrictions translate to certain responsibilities for
you if you distribute copies of the library or if you modify it.
For example, if you distribute copies of the library, whether gratis
or for a fee, you must give the recipients all the rights that we gave
you. You must make sure that they, too, receive or can get the source
code. If you link other code with the library, you must provide
complete object files to the recipients, so that they can relink them
with the library after making changes to the library and recompiling
it. And you must show them these terms so they know their rights.
We protect your rights with a two-step method: (1) we copyright the
library, and (2) we offer you this license, which gives you legal
permission to copy, distribute and/or modify the library.
To protect each distributor, we want to make it very clear that
there is no warranty for the free library. Also, if the library is
modified by someone else and passed on, the recipients should know
that what they have is not the original version, so that the original
author's reputation will not be affected by problems that might be
introduced by others.
Finally, software patents pose a constant threat to the existence of
any free program. We wish to make sure that a company cannot
effectively restrict the users of a free program by obtaining a
restrictive license from a patent holder. Therefore, we insist that
any patent license obtained for a version of the library must be
consistent with the full freedom of use specified in this license.
Most GNU software, including some libraries, is covered by the
ordinary GNU General Public License. This license, the GNU Lesser
General Public License, applies to certain designated libraries, and
is quite different from the ordinary General Public License. We use
this license for certain libraries in order to permit linking those
libraries into non-free programs.
When a program is linked with a library, whether statically or using
a shared library, the combination of the two is legally speaking a
combined work, a derivative of the original library. The ordinary
General Public License therefore permits such linking only if the
entire combination fits its criteria of freedom. The Lesser General
Public License permits more lax criteria for linking other code with
the library.
We call this license the "Lesser" General Public License because it
does Less to protect the user's freedom than the ordinary General
Public License. It also provides other free software developers Less
of an advantage over competing non-free programs. These disadvantages
are the reason we use the ordinary General Public License for many
libraries. However, the Lesser license provides advantages in certain
special circumstances.
For example, on rare occasions, there may be a special need to
encourage the widest possible use of a certain library, so that it becomes
a de-facto standard. To achieve this, non-free programs must be
allowed to use the library. A more frequent case is that a free
library does the same job as widely used non-free libraries. In this
case, there is little to gain by limiting the free library to free
software only, so we use the Lesser General Public License.
In other cases, permission to use a particular library in non-free
programs enables a greater number of people to use a large body of
free software. For example, permission to use the GNU C Library in
non-free programs enables many more people to use the whole GNU
operating system, as well as its variant, the GNU/Linux operating
system.
Although the Lesser General Public License is Less protective of the
users' freedom, it does ensure that the user of a program that is
linked with the Library has the freedom and the wherewithal to run
that program using a modified version of the Library.
The precise terms and conditions for copying, distribution and
modification follow. Pay close attention to the difference between a
"work based on the library" and a "work that uses the library". The
former contains code derived from the library, whereas the latter must
be combined with the library in order to run.
GNU LESSER GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License Agreement applies to any software library or other
program which contains a notice placed by the copyright holder or
other authorized party saying it may be distributed under the terms of
this Lesser General Public License (also called "this License").
Each licensee is addressed as "you".
A "library" means a collection of software functions and/or data
prepared so as to be conveniently linked with application programs
(which use some of those functions and data) to form executables.
The "Library", below, refers to any such software library or work
which has been distributed under these terms. A "work based on the
Library" means either the Library or any derivative work under
copyright law: that is to say, a work containing the Library or a
portion of it, either verbatim or with modifications and/or translated
straightforwardly into another language. (Hereinafter, translation is
included without limitation in the term "modification".)
"Source code" for a work means the preferred form of the work for
making modifications to it. For a library, complete source code means
all the source code for all modules it contains, plus any associated
interface definition files, plus the scripts used to control compilation
and installation of the library.
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running a program using the Library is not restricted, and output from
such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
appropriate copyright notice and disclaimer of warranty; keep intact
all the notices that refer to this License and to the absence of any
warranty; and distribute a copy of this License along with the
Library.
You may charge a fee for the physical act of transferring a copy,
and you may at your option offer warranty protection in exchange for a
fee.
2. You may modify your copy or copies of the Library or any portion
of it, thus forming a work based on the Library, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) The modified work must itself be a software library.
b) You must cause the files modified to carry prominent notices
stating that you changed the files and the date of any change.
c) You must cause the whole of the work to be licensed at no
charge to all third parties under the terms of this License.
d) If a facility in the modified Library refers to a function or a
table of data to be supplied by an application program that uses
the facility, other than as an argument passed when the facility
is invoked, then you must make a good faith effort to ensure that,
in the event an application does not supply such function or
table, the facility still operates, and performs whatever part of
its purpose remains meaningful.
(For example, a function in a library to compute square roots has
a purpose that is entirely well-defined independent of the
application. Therefore, Subsection 2d requires that any
application-supplied function or table used by this function must
be optional: if the application does not supply it, the square
root function must still compute square roots.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Library,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Library, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote
it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Library.
In addition, mere aggregation of another work not based on the Library
with the Library (or with a work based on the Library) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library. To do
this, you must alter all the notices that refer to this License, so
that they refer to the ordinary GNU General Public License, version 2,
instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.) Do not make any other change in
these notices.
Once this change is made in a given copy, it is irreversible for
that copy, so the ordinary GNU General Public License applies to all
subsequent copies and derivative works made from that copy.
This option is useful when you wish to copy part of the code of
the Library into a program that is not a library.
4. You may copy and distribute the Library (or a portion or
derivative of it, under Section 2) in object code or executable form
under the terms of Sections 1 and 2 above provided that you accompany
it with the complete corresponding machine-readable source code, which
must be distributed under the terms of Sections 1 and 2 above on a
medium customarily used for software interchange.
If distribution of object code is made by offering access to copy
from a designated place, then offering equivalent access to copy the
source code from the same place satisfies the requirement to
distribute the source code, even though third parties are not
compelled to copy the source along with the object code.
5. A program that contains no derivative of any portion of the
Library, but is designed to work with the Library by being compiled or
linked with it, is called a "work that uses the Library". Such a
work, in isolation, is not a derivative work of the Library, and
therefore falls outside the scope of this License.
However, linking a "work that uses the Library" with the Library
creates an executable that is a derivative of the Library (because it
contains portions of the Library), rather than a "work that uses the
library". The executable is therefore covered by this License.
Section 6 states terms for distribution of such executables.
When a "work that uses the Library" uses material from a header file
that is part of the Library, the object code for the work may be a
derivative work of the Library even though the source code is not.
Whether this is true is especially significant if the work can be
linked without the Library, or if the work is itself a library. The
threshold for this to be true is not precisely defined by law.
If such an object file uses only numerical parameters, data
structure layouts and accessors, and small macros and small inline
functions (ten lines or less in length), then the use of the object
file is unrestricted, regardless of whether it is legally a derivative
work. (Executables containing this object code plus portions of the
Library will still fall under Section 6.)
Otherwise, if the work is a derivative of the Library, you may
distribute the object code for the work under the terms of Section 6.
Any executables containing that work also fall under Section 6,
whether or not they are linked directly with the Library itself.
6. As an exception to the Sections above, you may also combine or
link a "work that uses the Library" with the Library to produce a
work containing portions of the Library, and distribute that work
under terms of your choice, provided that the terms permit
modification of the work for the customer's own use and reverse
engineering for debugging such modifications.
You must give prominent notice with each copy of the work that the
Library is used in it and that the Library and its use are covered by
this License. You must supply a copy of this License. If the work
during execution displays copyright notices, you must include the
copyright notice for the Library among them, as well as a reference
directing the user to the copy of this License. Also, you must do one
of these things:
a) Accompany the work with the complete corresponding
machine-readable source code for the Library including whatever
changes were used in the work (which must be distributed under
Sections 1 and 2 above); and, if the work is an executable linked
with the Library, with the complete machine-readable "work that
uses the Library", as object code and/or source code, so that the
user can modify the Library and then relink to produce a modified
executable containing the modified Library. (It is understood
that the user who changes the contents of definitions files in the
Library will not necessarily be able to recompile the application
to use the modified definitions.)
b) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (1) uses at run time a
copy of the library already present on the user's computer system,
rather than copying library functions into the executable, and (2)
will operate properly with a modified version of the library, if
the user installs one, as long as the modified version is
interface-compatible with the version that the work was made with.
c) Accompany the work with a written offer, valid for at
least three years, to give the same user the materials
specified in Subsection 6a, above, for a charge no more
than the cost of performing this distribution.
d) If distribution of the work is made by offering access to copy
from a designated place, offer equivalent access to copy the above
specified materials from the same place.
e) Verify that the user has already received a copy of these
materials or that you have already sent this user a copy.
For an executable, the required form of the "work that uses the
Library" must include any data and utility programs needed for
reproducing the executable from it. However, as a special exception,
the materials to be distributed need not include anything that is
normally distributed (in either source or binary form) with the major
components (compiler, kernel, and so on) of the operating system on
which the executable runs, unless that component itself accompanies
the executable.
It may happen that this requirement contradicts the license
restrictions of other proprietary libraries that do not normally
accompany the operating system. Such a contradiction means you cannot
use both them and the Library together in an executable that you
distribute.
7. You may place library facilities that are a work based on the
Library side-by-side in a single library together with other library
facilities not covered by this License, and distribute such a combined
library, provided that the separate distribution of the work based on
the Library and of the other library facilities is otherwise
permitted, and provided that you do these two things:
a) Accompany the combined library with a copy of the same work
based on the Library, uncombined with any other library
facilities. This must be distributed under the terms of the
Sections above.
b) Give prominent notice with the combined library of the fact
that part of it is a work based on the Library, and explaining
where to find the accompanying uncombined form of the same work.
8. You may not copy, modify, sublicense, link with, or distribute
the Library except as expressly provided under this License. Any
attempt otherwise to copy, modify, sublicense, link with, or
distribute the Library is void, and will automatically terminate your
rights under this License. However, parties who have received copies,
or rights, from you under this License will not have their licenses
terminated so long as such parties remain in full compliance.
9. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Library or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Library (or any work based on the
Library), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Library or works based on it.
10. Each time you redistribute the Library (or any work based on the
Library), the recipient automatically receives a license from the
original licensor to copy, distribute, link with or modify the Library
subject to these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties with
this License.
11. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Library at all. For example, if a patent
license would not permit royalty-free redistribution of the Library by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Library.
If any portion of this section is held invalid or unenforceable under any
particular circumstance, the balance of the section is intended to apply,
and the section as a whole is intended to apply in other circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
12. If the distribution and/or use of the Library is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Library under this License may add
an explicit geographical distribution limitation excluding those countries,
so that distribution is permitted only in or among countries not thus
excluded. In such case, this License incorporates the limitation as if
written in the body of this License.
13. The Free Software Foundation may publish revised and/or new
versions of the Lesser General Public License from time to time.
Such new versions will be similar in spirit to the present version,
but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Library
specifies a version number of this License which applies to it and
"any later version", you have the option of following the terms and
conditions either of that version or of any later version published by
the Free Software Foundation. If the Library does not specify a
license version number, you may choose any version ever published by
the Free Software Foundation.
14. If you wish to incorporate parts of the Library into other free
programs whose distribution conditions are incompatible with these,
write to the author to ask for permission. For software which is
copyrighted by the Free Software Foundation, write to the Free
Software Foundation; we sometimes make exceptions for this. Our
decision will be guided by the two goals of preserving the free status
of all derivatives of our free software and of promoting the sharing
and reuse of software generally.
NO WARRANTY
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Libraries
If you develop a new library, and you want it to be of the greatest
possible use to the public, we recommend making it free software that
everyone can redistribute and change. You can do so by permitting
redistribution under these terms (or, alternatively, under the terms of the
ordinary General Public License).
To apply these terms, attach the following notices to the library. It is
safest to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least the
"copyright" line and a pointer to where the full notice is found.
<one line to give the library's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the library, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
<signature of Moe Ghoul>, 1 April 1990
Moe Ghoul, President of Vice
That's all there is to it!

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,281 @@
XZ Utils
========
0. Overview
1. Documentation
1.1. Overall documentation
1.2. Documentation for command-line tools
1.3. Documentation for liblzma
2. Version numbering
3. Reporting bugs
4. Translations
4.1. Testing translations
5. Other implementations of the .xz format
6. Contact information
0. Overview
-----------
XZ Utils provide a general-purpose data-compression library plus
command-line tools. The native file format is the .xz format, but
also the legacy .lzma format is supported. The .xz format supports
multiple compression algorithms, which are called "filters" in the
context of XZ Utils. The primary filter is currently LZMA2. With
typical files, XZ Utils create about 30 % smaller files than gzip.
To ease adapting support for the .xz format into existing applications
and scripts, the API of liblzma is somewhat similar to the API of the
popular zlib library. For the same reason, the command-line tool xz
has a command-line syntax similar to that of gzip.
When aiming for the highest compression ratio, the LZMA2 encoder uses
a lot of CPU time and may use, depending on the settings, even
hundreds of megabytes of RAM. However, in fast modes, the LZMA2 encoder
competes with bzip2 in compression speed, RAM usage, and compression
ratio.
LZMA2 is reasonably fast to decompress. It is a little slower than
gzip, but a lot faster than bzip2. Being fast to decompress means
that the .xz format is especially nice when the same file will be
decompressed very many times (usually on different computers), which
is the case e.g. when distributing software packages. In such
situations, it's not too bad if the compression takes some time,
since that needs to be done only once to benefit many people.
With some file types, combining (or "chaining") LZMA2 with an
additional filter can improve the compression ratio. A filter chain may
contain up to four filters, although usually only one or two are used.
For example, putting a BCJ (Branch/Call/Jump) filter before LZMA2
in the filter chain can improve compression ratio of executable files.
Since the .xz format allows adding new filter IDs, it is possible that
some day there will be a filter that is, for example, much faster to
compress than LZMA2 (but probably with worse compression ratio).
Similarly, it is possible that some day there is a filter that will
compress better than LZMA2.
XZ Utils supports multithreaded compression. XZ Utils doesn't support
multithreaded decompression yet. It has been planned though and taken
into account when designing the .xz file format. In the future, files
that were created in threaded mode can be decompressed in threaded
mode too.
1. Documentation
----------------
1.1. Overall documentation
README This file
INSTALL.generic Generic install instructions for those not
familiar with packages using GNU Autotools
INSTALL Installation instructions specific to XZ Utils
PACKAGERS Information to packagers of XZ Utils
COPYING XZ Utils copyright and license information
COPYING.0BSD BSD Zero Clause License
COPYING.GPLv2 GNU General Public License version 2
COPYING.GPLv3 GNU General Public License version 3
COPYING.LGPLv2.1 GNU Lesser General Public License version 2.1
AUTHORS The main authors of XZ Utils
THANKS Incomplete list of people who have helped making
this software
NEWS User-visible changes between XZ Utils releases
ChangeLog Detailed list of changes (commit log)
TODO Known bugs and some sort of to-do list
Note that only some of the above files are included in binary
packages.
1.2. Documentation for command-line tools
The command-line tools are documented as man pages. In source code
releases (and possibly also in some binary packages), the man pages
are also provided in plain text (ASCII only) format in the directory
"doc/man" to make the man pages more accessible to those whose
operating system doesn't provide an easy way to view man pages.
1.3. Documentation for liblzma
The liblzma API headers include short docs about each function
and data type as Doxygen tags. These docs should be quite OK as
a quick reference.
There are a few example/tutorial programs that should help in
getting started with liblzma. In the source package the examples
are in "doc/examples" and in binary packages they may be under
"examples" in the same directory as this README.
Since the liblzma API has similarities to the zlib API, some people
may find it useful to read the zlib docs and tutorial too:
https://zlib.net/manual.html
https://zlib.net/zlib_how.html
2. Version numbering
--------------------
The version number format of XZ Utils is X.Y.ZS:
- X is the major version. When this is incremented, the library
API and ABI break.
- Y is the minor version. It is incremented when new features
are added without breaking the existing API or ABI. An even Y
indicates a stable release and an odd Y indicates unstable
(alpha or beta version).
- Z is the revision. This has a different meaning for stable and
unstable releases:
* Stable: Z is incremented when bugs get fixed without adding
any new features. This is intended to be convenient for
downstream distributors that want bug fixes but don't want
any new features to minimize the risk of introducing new bugs.
* Unstable: Z is just a counter. API or ABI of features added
in earlier unstable releases having the same X.Y may break.
- S indicates stability of the release. It is missing from the
stable releases, where Y is an even number. When Y is odd, S
is either "alpha" or "beta" to make it very clear that such
versions are not stable releases. The same X.Y.Z combination is
not used for more than one stability level, i.e. after X.Y.Zalpha,
the next version can be X.Y.(Z+1)beta but not X.Y.Zbeta.
3. Reporting bugs
-----------------
Naturally it is easiest for me if you already know what causes the
unexpected behavior. Even better if you have a patch to propose.
However, quite often the reason for unexpected behavior is unknown,
so here are a few things to do before sending a bug report:
1. Try to create a small example how to reproduce the issue.
2. Compile XZ Utils with debugging code using configure switches
--enable-debug and, if possible, --disable-shared. If you are
using GCC, use CFLAGS='-O0 -ggdb3'. Don't strip the resulting
binaries.
3. Turn on core dumps. The exact command depends on your shell;
for example in GNU bash it is done with "ulimit -c unlimited",
and in tcsh with "limit coredumpsize unlimited".
4. Try to reproduce the suspected bug. If you get "assertion failed"
message, be sure to include the complete message in your bug
report. If the application leaves a coredump, get a backtrace
using gdb:
$ gdb /path/to/app-binary # Load the app to the debugger.
(gdb) core core # Open the coredump.
(gdb) bt # Print the backtrace. Copy & paste to bug report.
(gdb) quit # Quit gdb.
Report your bug via email or IRC (see Contact information below).
Don't send core dump files or any executables. If you have a small
example file(s) (total size less than 256 KiB), please include
it/them as an attachment. If you have bigger test files, put them
online somewhere and include a URL to the file(s) in the bug report.
Always include the exact version number of XZ Utils in the bug report.
If you are using a snapshot from the git repository, use "git describe"
to get the exact snapshot version. If you are using XZ Utils shipped
in an operating system distribution, mention the distribution name,
distribution version, and exact xz package version; if you cannot
repeat the bug with the code compiled from unpatched source code,
you probably need to report a bug to your distribution's bug tracking
system.
4. Translations
---------------
The xz command line tool and all man pages can be translated.
The translations are handled via the Translation Project. If you
wish to help translating xz, please join the Translation Project:
https://translationproject.org/html/translators.html
Updates to translations won't be accepted by methods that bypass
the Translation Project because there is a risk of duplicate work:
translation updates made in the xz repository aren't seen by the
translators in the Translation Project. If you have found bugs in
a translation, please report them to the Language-Team address
which can be found near the beginning of the PO file.
If you find language problems in the original English strings,
feel free to suggest improvements. Ask if something is unclear.
4.1. Testing translations
Testing can be done by installing xz into a temporary directory.
If building from Git repository (not tarball), generate the
Autotools files:
./autogen.sh
Create a subdirectory for the build files. The tmp-build directory
can be deleted after testing.
mkdir tmp-build
cd tmp-build
../configure --disable-shared --enable-debug --prefix=$PWD/inst
Edit the .po file in the po directory. Then build and install to
the "tmp-build/inst" directory, and use translations.bash to see
how some of the messages look. Repeat these steps if needed:
make -C po update-po
make -j"$(nproc)" install
bash ../debug/translation.bash | less
bash ../debug/translation.bash | less -S # For --list outputs
To test other languages, set the LANGUAGE environment variable
before running translations.bash. The value should match the PO file
name without the .po suffix. Example:
export LANGUAGE=fi
5. Other implementations of the .xz format
------------------------------------------
7-Zip and the p7zip port of 7-Zip support the .xz format starting
from the version 9.00alpha.
https://7-zip.org/
https://p7zip.sourceforge.net/
XZ Embedded is a limited implementation written for use in the Linux
kernel, but it is also suitable for other embedded use.
https://tukaani.org/xz/embedded.html
XZ for Java is a complete implementation written in pure Java.
https://tukaani.org/xz/java.html
6. Contact information
----------------------
XZ Utils in general:
- Home page: https://tukaani.org/xz/
- Email to maintainer(s): xz@tukaani.org
- IRC: #tukaani on Libera Chat
- GitHub: https://github.com/tukaani-project/xz
Lead maintainer:
- Email: Lasse Collin <lasse.collin@tukaani.org>
- IRC: Larhzu on Libera Chat

View file

@ -0,0 +1,262 @@
Thanks
======
Some people have helped more, some less, but nevertheless everyone's help
has been important. :-)
- Adam Borowski
- Adam Walling
- Adrien Nader
- Agostino Sarubbo
- Alexander Bluhm
- Alexander M. Greenham
- Alexander Neumann
- Alexandre Sauvé
- Alexey Tourbin
- Anders F. Björklund
- Andraž 'ruskie' Levstik
- Andre Noll
- Andreas K. Hüttel
- Andreas Müller
- Andreas Schwab
- Andreas Zieringer
- Andrej Skenderija
- Andres Freund
- Andrew Dudman
- Andrew Murray
- Antoine Cœur
- Anton Kochkov
- Antonio Diaz Diaz
- Arkadiusz Miskiewicz
- Asgeir Storesund Nilsen
- Aziz Chaudhry
- Bela Lubkin
- Ben Boeckel
- Benjamin Buch
- Benno Schulenberg
- Bernhard Reutner-Fischer
- Bert Wesarg
- Bhargava Shastry
- Bill Glessner
- Bjarni Ingi Gislason
- Boud Roukema
- Brad Smith
- Bruce Stark
- Cary Lewis
- Charles Wilson
- Chase Knowlden
- Chenxi Mao
- Chien Wong
- Chris Donawa
- Chris McCrohan
- Christian Hesse
- Christian Kujau
- Christian von Roques
- Christian Weisgerber
- Christoph Junghans
- Collin Funk
- Conley Moorhous
- Cristian Rodríguez
- Cristiano Ceglia
- Dan Shechter
- Dan Stromberg
- Dan Weiss
- Daniel Leonard
- Daniel Mealha Cabrita
- Daniel Packard
- Daniel Richard G.
- David Burklund
- Denis Excoffier
- Derwin McGeary
- Dexter Castor Döpping
- Diederik de Haas
- Diego Elio Pettenò
- Dimitri Papadopoulos Orfanos
- Dirk Müller
- Douglas Thor
- Ed Maste
- Elbert Pol
- Eli Schwartz
- Elijah Almeida Coimbra
- Émilie Labbé
- Emmanuel Blot
- Eric Lindblad
- Eric S. Raymond
- Étienne Mollier
- Evan Nemerson
- Fangrui Song
- Felix Collin
- Filip Palian
- Firas Khalil Khana
- François Etcheverry
- Frank Busse
- Frank Prochnow
- Fredrik Wikstrom
- Gabi Davar
- Gabriela Gutierrez
- Gilles Espinasse
- Gregory Margo
- Guillaume Outters
- Guiorgy Potskhishvili
- H. Peter Anvin
- Hajin Jang
- Hans Jansen
- Harri K. Koskinen
- Hin-Tak Leung
- H.J. Lu
- Hongbo Ni
- Igor Pavlov
- İhsan Doğan
- Ilya Kurdyukov
- Iouri Kharon
- İsmail Dönmez
- Ivan A. Melnikov
- Jakub Bogusz
- James Buren
- James M Leddy
- Jan Kratochvil
- Jan Terje Hansen
- Jason Gorski
- Jeff Bastian
- Jeffrey Walton
- Jeroen Roovers
- Jim Meyering
- Jim Wilcoxson
- Joachim Henke
- John Paul Adrian Glaubitz
- Jonathan Nieder
- Jonathan Stott
- Joona Kannisto
- Jouk Jansen
- Juan Manuel Guerrero
- Jukka Salmi
- Julien Marrec
- Jun I Jin
- Kai Pastor
- Karl Beldan
- Karl Berry
- Keith Patton
- Kelvin Lee
- Kevin R. Bulgrien
- Kian-Meng Ang
- Kim Jinyeong
- Kirill A. Korinsky
- Kiyoshi Kanazawa
- Lars Wirzenius
- Li Chenggang
- Lizandro Heredia
- Loganaden Velvindron
- Lorenzo De Liso
- Lukas Braune
- Maarten Bosmans
- Maksym Vatsyk
- Marcin Kowalczyk
- Marcus Comstedt
- Marcus Tillmanns
- Marek Černocký
- Mark Adler
- Mark Wielaard
- Markus Duft
- Markus Rickert
- Martin Blumenstingl
- Martin Matuška
- Martin Storsjö
- Martin Väth
- Mathieu Vachon
- Matthew Good
- Matthieu Rakotojaona
- Melanie Blower
- Michael Felt
- Michael Fox
- Michał Górny
- Mike Frysinger
- Mikko Pouru
- Milo Casagrande
- Mohammed Adnène Trojette
- Nathan Moinvaziri
- Nelson H. F. Beebe
- Nicholas Jackson
- Nobuhiro Iwamatsu
- Ole André Vadla Ravnås
- Orange Tsai
- Orgad Shaneh
- Patrick J. Volkerding
- Paul Eggert
- Paul Townsend
- Pavel Raiskup
- Per Øyvind Karlsen
- Peter Ivanov
- Peter Lawler
- Peter O'Gorman
- Peter Pallinger
- Peter Seiderer
- Pierre-Yves Martin
- Pilorz Wojciech
- Pino Toscano
- Pippijn van Steenhoven
- Radek Zikmund
- Rafał Mużyło
- Rainer Müller
- Ralf Wildenhues
- Rich Prohaska
- Richard Koch
- Richard W.M. Jones
- Robert Elz
- Robert Readman
- Roel Bouckaert
- Ron Desmond
- Ruarí Ødegaard
- Rui Paulo
- Ryan Colyer
- Ryan Young
- Salman Muin Kayser Chishti
- Sam James
- Scott McAllister
- Sean Fenian
- Sebastian Andrzej Siewior
- Sergey Kosukhin
- Simon Josefsson
- Siteshwar Vashisht
- Steffen Nurpmeso
- Stephan Kulow
- Stephen Sachs
- Stuart Shelton
- Taiki Tsunekawa
- Thomas Klausner
- Tobias Lahrmann Hansen
- Tobias Stoeckmann
- Tomasz Gajc
- Tomer Chachamu
- Torsten Rupp
- Trần Ngọc Quân
- Trent W. Buck
- Victoria Alexia
- Vijay Sarvepalli
- Ville Koskinen
- Ville Skyttä
- Vincent Cruz
- Vincent Fazio
- Vincent Lefevre
- Vincent Torri
- Vincent Wixsom
- Vincenzo Innocente
- Vitaly Chikunov
- Wim Lewis
- Xi Ruoyao
- Xin Li
- Yifeng Li
- 榆柳松 (ZhengSen Wang)
Companies:
- Google
- Sandfly Security
Other credits:
- cleemy desu wayo working with Trend Micro Zero Day Initiative
- Orange Tsai and splitline from DEVCORE Research Team
Also thanks to all the people who have participated in the Tukaani project.
I have probably forgot to add some names to the above list. Sorry about
that and thanks for your help.

View file

@ -0,0 +1,88 @@
XZ Utils To-Do List
===================
Known bugs
----------
The test suite is incomplete.
XZ Utils compress some files significantly worse than LZMA Utils.
This is due to faster compression presets used by XZ Utils, and
can often be worked around by using "xz --extreme". With some files
--extreme isn't enough though: it's most likely with files that
compress extremely well, so going from compression ratio of 0.003
to 0.004 means big relative increase in the compressed file size.
tuklib_exit() doesn't block signals => EINTR is possible.
If liblzma has created threads and fork() gets called, liblzma
code will break in the child process unless it calls exec() and
doesn't touch liblzma.
Missing features
----------------
Add support for storing metadata in .xz files. A preliminary
idea is to create a new Stream type for metadata. When both
metadata and data are wanted in the same .xz file, two or more
Streams would be concatenated.
The state stored in lzma_stream should be cloneable, which would
be mostly useful when using a preset dictionary in LZMA2, but
it may have other uses too. Compare to deflateCopy() in zlib.
Adjust dictionary size when the input file size is known.
Maybe do this only if an option is given.
xz doesn't support copying extended attributes, access control
lists etc. from source to target file.
Multithreaded compression:
- Reduce memory usage of the current method.
- Implement threaded match finders.
- Implement pigz-style threading in LZMA2.
Buffer-to-buffer coding could use less RAM (especially when
decompressing LZMA1 or LZMA2).
I/O library is not implemented (similar to gzopen() in zlib).
It will be a separate library that supports uncompressed, .gz,
.bz2, .lzma, and .xz files.
Support changing lzma_options_lzma.mode with lzma_filters_update().
Support LZMA_FULL_FLUSH for lzma_stream_decoder() to stop at
Block and Stream boundaries.
Error codes from lzma_code() aren't very specific. A more detailed
error message (string) could be provided too. It could be returned
by a new function or use a currently-reserved member of lzma_stream.
Make it possible to adjust LZMA2 options in the middle of a Block
so that the encoding speed vs. compression ratio can be optimized
when the compressed data is streamed over network.
Improved BCJ filters. The current filters are small but they aren't
so great when compressing binary packages that contain various file
types. Specifically, they make things worse if there are static
libraries or Linux kernel modules. The filtering could also be
more effective (without getting overly complex), for example,
streamable variant BCJ2 from 7-Zip could be implemented.
Filter that autodetects specific data types in the input stream
and applies appropriate filters for the corrects parts of the input.
Perhaps combine this with the BCJ filter improvement point above.
Long-range LZ77 method as a separate filter or as a new LZMA2
match finder.
Documentation
-------------
More tutorial programs are needed for liblzma.
Document the LZMA1 and LZMA2 algorithms.

View file

@ -0,0 +1,31 @@
liblzma example programs
========================
Introduction
The examples are written so that the same comments aren't
repeated (much) in later files.
On POSIX systems, the examples should build by just typing "make".
The examples that use stdin or stdout don't set stdin and stdout
to binary mode. On systems where it matters (e.g. Windows) it is
possible that the examples won't work without modification.
List of examples
01_compress_easy.c Multi-call compression using
a compression preset
02_decompress.c Multi-call decompression
03_compress_custom.c Like 01_compress_easy.c but using
a custom filter chain
(x86 BCJ + LZMA2)
04_compress_easy_mt.c Multi-threaded multi-call
compression using a compression
preset

View file

@ -0,0 +1,296 @@
// SPDX-License-Identifier: 0BSD
///////////////////////////////////////////////////////////////////////////////
//
/// \file 01_compress_easy.c
/// \brief Compress from stdin to stdout in multi-call mode
///
/// Usage: ./01_compress_easy PRESET < INFILE > OUTFILE
///
/// Example: ./01_compress_easy 6 < foo > foo.xz
//
// Author: Lasse Collin
//
///////////////////////////////////////////////////////////////////////////////
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <lzma.h>
static void
show_usage_and_exit(const char *argv0)
{
fprintf(stderr, "Usage: %s PRESET < INFILE > OUTFILE\n"
"PRESET is a number 0-9 and can optionally be "
"followed by 'e' to indicate extreme preset\n",
argv0);
exit(EXIT_FAILURE);
}
static uint32_t
get_preset(int argc, char **argv)
{
// One argument whose first char must be 0-9.
if (argc != 2 || argv[1][0] < '0' || argv[1][0] > '9')
show_usage_and_exit(argv[0]);
// Calculate the preste level 0-9.
uint32_t preset = argv[1][0] - '0';
// If there is a second char, it must be 'e'. It will set
// the LZMA_PRESET_EXTREME flag.
if (argv[1][1] != '\0') {
if (argv[1][1] != 'e' || argv[1][2] != '\0')
show_usage_and_exit(argv[0]);
preset |= LZMA_PRESET_EXTREME;
}
return preset;
}
static bool
init_encoder(lzma_stream *strm, uint32_t preset)
{
// Initialize the encoder using a preset. Set the integrity to check
// to CRC64, which is the default in the xz command line tool. If
// the .xz file needs to be decompressed with XZ Embedded, use
// LZMA_CHECK_CRC32 instead.
lzma_ret ret = lzma_easy_encoder(strm, preset, LZMA_CHECK_CRC64);
// Return successfully if the initialization went fine.
if (ret == LZMA_OK)
return true;
// Something went wrong. The possible errors are documented in
// lzma/container.h (src/liblzma/api/lzma/container.h in the source
// package or e.g. /usr/include/lzma/container.h depending on the
// install prefix).
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_OPTIONS_ERROR:
msg = "Specified preset is not supported";
break;
case LZMA_UNSUPPORTED_CHECK:
msg = "Specified integrity check is not supported";
break;
default:
// This is most likely LZMA_PROG_ERROR indicating a bug in
// this program or in liblzma. It is inconvenient to have a
// separate error message for errors that should be impossible
// to occur, but knowing the error code is important for
// debugging. That's why it is good to print the error code
// at least when there is no good error message to show.
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Error initializing the encoder: %s (error code %u)\n",
msg, ret);
return false;
}
static bool
compress(lzma_stream *strm, FILE *infile, FILE *outfile)
{
// This will be LZMA_RUN until the end of the input file is reached.
// This tells lzma_code() when there will be no more input.
lzma_action action = LZMA_RUN;
// Buffers to temporarily hold uncompressed input
// and compressed output.
uint8_t inbuf[BUFSIZ];
uint8_t outbuf[BUFSIZ];
// Initialize the input and output pointers. Initializing next_in
// and avail_in isn't really necessary when we are going to encode
// just one file since LZMA_STREAM_INIT takes care of initializing
// those already. But it doesn't hurt much and it will be needed
// if encoding more than one file like we will in 02_decompress.c.
//
// While we don't care about strm->total_in or strm->total_out in this
// example, it is worth noting that initializing the encoder will
// always reset total_in and total_out to zero. But the encoder
// initialization doesn't touch next_in, avail_in, next_out, or
// avail_out.
strm->next_in = NULL;
strm->avail_in = 0;
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
// Loop until the file has been successfully compressed or until
// an error occurs.
while (true) {
// Fill the input buffer if it is empty.
if (strm->avail_in == 0 && !feof(infile)) {
strm->next_in = inbuf;
strm->avail_in = fread(inbuf, 1, sizeof(inbuf),
infile);
if (ferror(infile)) {
fprintf(stderr, "Read error: %s\n",
strerror(errno));
return false;
}
// Once the end of the input file has been reached,
// we need to tell lzma_code() that no more input
// will be coming and that it should finish the
// encoding.
if (feof(infile))
action = LZMA_FINISH;
}
// Tell liblzma do the actual encoding.
//
// This reads up to strm->avail_in bytes of input starting
// from strm->next_in. avail_in will be decremented and
// next_in incremented by an equal amount to match the
// number of input bytes consumed.
//
// Up to strm->avail_out bytes of compressed output will be
// written starting from strm->next_out. avail_out and next_out
// will be incremented by an equal amount to match the number
// of output bytes written.
//
// The encoder has to do internal buffering, which means that
// it may take quite a bit of input before the same data is
// available in compressed form in the output buffer.
lzma_ret ret = lzma_code(strm, action);
// If the output buffer is full or if the compression finished
// successfully, write the data from the output buffer to
// the output file.
if (strm->avail_out == 0 || ret == LZMA_STREAM_END) {
// When lzma_code() has returned LZMA_STREAM_END,
// the output buffer is likely to be only partially
// full. Calculate how much new data there is to
// be written to the output file.
size_t write_size = sizeof(outbuf) - strm->avail_out;
if (fwrite(outbuf, 1, write_size, outfile)
!= write_size) {
fprintf(stderr, "Write error: %s\n",
strerror(errno));
return false;
}
// Reset next_out and avail_out.
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
}
// Normally the return value of lzma_code() will be LZMA_OK
// until everything has been encoded.
if (ret != LZMA_OK) {
// Once everything has been encoded successfully, the
// return value of lzma_code() will be LZMA_STREAM_END.
//
// It is important to check for LZMA_STREAM_END. Do not
// assume that getting ret != LZMA_OK would mean that
// everything has gone well.
if (ret == LZMA_STREAM_END)
return true;
// It's not LZMA_OK nor LZMA_STREAM_END,
// so it must be an error code. See lzma/base.h
// (src/liblzma/api/lzma/base.h in the source package
// or e.g. /usr/include/lzma/base.h depending on the
// install prefix) for the list and documentation of
// possible values. Most values listen in lzma_ret
// enumeration aren't possible in this example.
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_DATA_ERROR:
// This error is returned if the compressed
// or uncompressed size get near 8 EiB
// (2^63 bytes) because that's where the .xz
// file format size limits currently are.
// That is, the possibility of this error
// is mostly theoretical unless you are doing
// something very unusual.
//
// Note that strm->total_in and strm->total_out
// have nothing to do with this error. Changing
// those variables won't increase or decrease
// the chance of getting this error.
msg = "File size limits exceeded";
break;
default:
// This is most likely LZMA_PROG_ERROR, but
// if this program is buggy (or liblzma has
// a bug), it may be e.g. LZMA_BUF_ERROR or
// LZMA_OPTIONS_ERROR too.
//
// It is inconvenient to have a separate
// error message for errors that should be
// impossible to occur, but knowing the error
// code is important for debugging. That's why
// it is good to print the error code at least
// when there is no good error message to show.
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Encoder error: %s (error code %u)\n",
msg, ret);
return false;
}
}
}
extern int
main(int argc, char **argv)
{
// Get the preset number from the command line.
uint32_t preset = get_preset(argc, argv);
// Initialize a lzma_stream structure. When it is allocated on stack,
// it is simplest to use LZMA_STREAM_INIT macro like below. When it
// is allocated on heap, using memset(strmptr, 0, sizeof(*strmptr))
// works (as long as NULL pointers are represented with zero bits
// as they are on practically all computers today).
lzma_stream strm = LZMA_STREAM_INIT;
// Initialize the encoder. If it succeeds, compress from
// stdin to stdout.
bool success = init_encoder(&strm, preset);
if (success)
success = compress(&strm, stdin, stdout);
// Free the memory allocated for the encoder. If we were encoding
// multiple files, this would only need to be done after the last
// file. See 02_decompress.c for handling of multiple files.
//
// It is OK to call lzma_end() multiple times or when it hasn't been
// actually used except initialized with LZMA_STREAM_INIT.
lzma_end(&strm);
// Close stdout to catch possible write errors that can occur
// when pending data is flushed from the stdio buffers.
if (fclose(stdout)) {
fprintf(stderr, "Write error: %s\n", strerror(errno));
success = false;
}
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}

View file

@ -0,0 +1,286 @@
// SPDX-License-Identifier: 0BSD
///////////////////////////////////////////////////////////////////////////////
//
/// \file 02_decompress.c
/// \brief Decompress .xz files to stdout
///
/// Usage: ./02_decompress INPUT_FILES... > OUTFILE
///
/// Example: ./02_decompress foo.xz bar.xz > foobar
//
// Author: Lasse Collin
//
///////////////////////////////////////////////////////////////////////////////
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <lzma.h>
static bool
init_decoder(lzma_stream *strm)
{
// Initialize a .xz decoder. The decoder supports a memory usage limit
// and a set of flags.
//
// The memory usage of the decompressor depends on the settings used
// to compress a .xz file. It can vary from less than a megabyte to
// a few gigabytes, but in practice (at least for now) it rarely
// exceeds 65 MiB because that's how much memory is required to
// decompress files created with "xz -9". Settings requiring more
// memory take extra effort to use and don't (at least for now)
// provide significantly better compression in most cases.
//
// Memory usage limit is useful if it is important that the
// decompressor won't consume gigabytes of memory. The need
// for limiting depends on the application. In this example,
// no memory usage limiting is used. This is done by setting
// the limit to UINT64_MAX.
//
// The .xz format allows concatenating compressed files as is:
//
// echo foo | xz > foobar.xz
// echo bar | xz >> foobar.xz
//
// When decompressing normal standalone .xz files, LZMA_CONCATENATED
// should always be used to support decompression of concatenated
// .xz files. If LZMA_CONCATENATED isn't used, the decoder will stop
// after the first .xz stream. This can be useful when .xz data has
// been embedded inside another file format.
//
// Flags other than LZMA_CONCATENATED are supported too, and can
// be combined with bitwise-or. See lzma/container.h
// (src/liblzma/api/lzma/container.h in the source package or e.g.
// /usr/include/lzma/container.h depending on the install prefix)
// for details.
lzma_ret ret = lzma_stream_decoder(
strm, UINT64_MAX, LZMA_CONCATENATED);
// Return successfully if the initialization went fine.
if (ret == LZMA_OK)
return true;
// Something went wrong. The possible errors are documented in
// lzma/container.h (src/liblzma/api/lzma/container.h in the source
// package or e.g. /usr/include/lzma/container.h depending on the
// install prefix).
//
// Note that LZMA_MEMLIMIT_ERROR is never possible here. If you
// specify a very tiny limit, the error will be delayed until
// the first headers have been parsed by a call to lzma_code().
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_OPTIONS_ERROR:
msg = "Unsupported decompressor flags";
break;
default:
// This is most likely LZMA_PROG_ERROR indicating a bug in
// this program or in liblzma. It is inconvenient to have a
// separate error message for errors that should be impossible
// to occur, but knowing the error code is important for
// debugging. That's why it is good to print the error code
// at least when there is no good error message to show.
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Error initializing the decoder: %s (error code %u)\n",
msg, ret);
return false;
}
static bool
decompress(lzma_stream *strm, const char *inname, FILE *infile, FILE *outfile)
{
// When LZMA_CONCATENATED flag was used when initializing the decoder,
// we need to tell lzma_code() when there will be no more input.
// This is done by setting action to LZMA_FINISH instead of LZMA_RUN
// in the same way as it is done when encoding.
//
// When LZMA_CONCATENATED isn't used, there is no need to use
// LZMA_FINISH to tell when all the input has been read, but it
// is still OK to use it if you want. When LZMA_CONCATENATED isn't
// used, the decoder will stop after the first .xz stream. In that
// case some unused data may be left in strm->next_in.
lzma_action action = LZMA_RUN;
uint8_t inbuf[BUFSIZ];
uint8_t outbuf[BUFSIZ];
strm->next_in = NULL;
strm->avail_in = 0;
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
while (true) {
if (strm->avail_in == 0 && !feof(infile)) {
strm->next_in = inbuf;
strm->avail_in = fread(inbuf, 1, sizeof(inbuf),
infile);
if (ferror(infile)) {
fprintf(stderr, "%s: Read error: %s\n",
inname, strerror(errno));
return false;
}
// Once the end of the input file has been reached,
// we need to tell lzma_code() that no more input
// will be coming. As said before, this isn't required
// if the LZMA_CONCATENATED flag isn't used when
// initializing the decoder.
if (feof(infile))
action = LZMA_FINISH;
}
lzma_ret ret = lzma_code(strm, action);
if (strm->avail_out == 0 || ret == LZMA_STREAM_END) {
size_t write_size = sizeof(outbuf) - strm->avail_out;
if (fwrite(outbuf, 1, write_size, outfile)
!= write_size) {
fprintf(stderr, "Write error: %s\n",
strerror(errno));
return false;
}
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
}
if (ret != LZMA_OK) {
// Once everything has been decoded successfully, the
// return value of lzma_code() will be LZMA_STREAM_END.
//
// It is important to check for LZMA_STREAM_END. Do not
// assume that getting ret != LZMA_OK would mean that
// everything has gone well or that when you aren't
// getting more output it must have successfully
// decoded everything.
if (ret == LZMA_STREAM_END)
return true;
// It's not LZMA_OK nor LZMA_STREAM_END,
// so it must be an error code. See lzma/base.h
// (src/liblzma/api/lzma/base.h in the source package
// or e.g. /usr/include/lzma/base.h depending on the
// install prefix) for the list and documentation of
// possible values. Many values listen in lzma_ret
// enumeration aren't possible in this example, but
// can be made possible by enabling memory usage limit
// or adding flags to the decoder initialization.
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_FORMAT_ERROR:
// .xz magic bytes weren't found.
msg = "The input is not in the .xz format";
break;
case LZMA_OPTIONS_ERROR:
// For example, the headers specify a filter
// that isn't supported by this liblzma
// version (or it hasn't been enabled when
// building liblzma, but no-one sane does
// that unless building liblzma for an
// embedded system). Upgrading to a newer
// liblzma might help.
//
// Note that it is unlikely that the file has
// accidentally became corrupt if you get this
// error. The integrity of the .xz headers is
// always verified with a CRC32, so
// unintentionally corrupt files can be
// distinguished from unsupported files.
msg = "Unsupported compression options";
break;
case LZMA_DATA_ERROR:
msg = "Compressed file is corrupt";
break;
case LZMA_BUF_ERROR:
// Typically this error means that a valid
// file has got truncated, but it might also
// be a damaged part in the file that makes
// the decoder think the file is truncated.
// If you prefer, you can use the same error
// message for this as for LZMA_DATA_ERROR.
msg = "Compressed file is truncated or "
"otherwise corrupt";
break;
default:
// This is most likely LZMA_PROG_ERROR.
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "%s: Decoder error: "
"%s (error code %u)\n",
inname, msg, ret);
return false;
}
}
}
extern int
main(int argc, char **argv)
{
if (argc <= 1) {
fprintf(stderr, "Usage: %s FILES...\n", argv[0]);
return EXIT_FAILURE;
}
lzma_stream strm = LZMA_STREAM_INIT;
bool success = true;
// Try to decompress all files.
for (int i = 1; i < argc; ++i) {
if (!init_decoder(&strm)) {
// Decoder initialization failed. There's no point
// to retry it so we need to exit.
success = false;
break;
}
FILE *infile = fopen(argv[i], "rb");
if (infile == NULL) {
fprintf(stderr, "%s: Error opening the "
"input file: %s\n",
argv[i], strerror(errno));
success = false;
} else {
success &= decompress(&strm, argv[i], infile, stdout);
fclose(infile);
}
}
// Free the memory allocated for the decoder. This only needs to be
// done after the last file.
lzma_end(&strm);
if (fclose(stdout)) {
fprintf(stderr, "Write error: %s\n", strerror(errno));
success = false;
}
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}

View file

@ -0,0 +1,192 @@
// SPDX-License-Identifier: 0BSD
///////////////////////////////////////////////////////////////////////////////
//
/// \file 03_compress_custom.c
/// \brief Compress in multi-call mode using x86 BCJ and LZMA2
///
/// Usage: ./03_compress_custom < INFILE > OUTFILE
///
/// Example: ./03_compress_custom < foo > foo.xz
//
// Author: Lasse Collin
//
///////////////////////////////////////////////////////////////////////////////
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <lzma.h>
static bool
init_encoder(lzma_stream *strm)
{
// Use the default preset (6) for LZMA2.
//
// The lzma_options_lzma structure and the lzma_lzma_preset() function
// are declared in lzma/lzma12.h (src/liblzma/api/lzma/lzma12.h in the
// source package or e.g. /usr/include/lzma/lzma12.h depending on
// the install prefix).
lzma_options_lzma opt_lzma2;
if (lzma_lzma_preset(&opt_lzma2, LZMA_PRESET_DEFAULT)) {
// It should never fail because the default preset
// (and presets 0-9 optionally with LZMA_PRESET_EXTREME)
// are supported by all stable liblzma versions.
//
// (The encoder initialization later in this function may
// still fail due to unsupported preset *if* the features
// required by the preset have been disabled at build time,
// but no-one does such things except on embedded systems.)
fprintf(stderr, "Unsupported preset, possibly a bug\n");
return false;
}
// Now we could customize the LZMA2 options if we wanted. For example,
// we could set the dictionary size (opt_lzma2.dict_size) to
// something else than the default (8 MiB) of the default preset.
// See lzma/lzma12.h for details of all LZMA2 options.
//
// The x86 BCJ filter will try to modify the x86 instruction stream so
// that LZMA2 can compress it better. The x86 BCJ filter doesn't need
// any options so it will be set to NULL below.
//
// Construct the filter chain. The uncompressed data goes first to
// the first filter in the array, in this case the x86 BCJ filter.
// The array is always terminated by setting .id = LZMA_VLI_UNKNOWN.
//
// See lzma/filter.h for more information about the lzma_filter
// structure.
lzma_filter filters[] = {
{ .id = LZMA_FILTER_X86, .options = NULL },
{ .id = LZMA_FILTER_LZMA2, .options = &opt_lzma2 },
{ .id = LZMA_VLI_UNKNOWN, .options = NULL },
};
// Initialize the encoder using the custom filter chain.
lzma_ret ret = lzma_stream_encoder(strm, filters, LZMA_CHECK_CRC64);
if (ret == LZMA_OK)
return true;
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_OPTIONS_ERROR:
// We are no longer using a plain preset so this error
// message has been edited accordingly compared to
// 01_compress_easy.c.
msg = "Specified filter chain is not supported";
break;
case LZMA_UNSUPPORTED_CHECK:
msg = "Specified integrity check is not supported";
break;
default:
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Error initializing the encoder: %s (error code %u)\n",
msg, ret);
return false;
}
// This function is identical to the one in 01_compress_easy.c.
static bool
compress(lzma_stream *strm, FILE *infile, FILE *outfile)
{
lzma_action action = LZMA_RUN;
uint8_t inbuf[BUFSIZ];
uint8_t outbuf[BUFSIZ];
strm->next_in = NULL;
strm->avail_in = 0;
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
while (true) {
if (strm->avail_in == 0 && !feof(infile)) {
strm->next_in = inbuf;
strm->avail_in = fread(inbuf, 1, sizeof(inbuf),
infile);
if (ferror(infile)) {
fprintf(stderr, "Read error: %s\n",
strerror(errno));
return false;
}
if (feof(infile))
action = LZMA_FINISH;
}
lzma_ret ret = lzma_code(strm, action);
if (strm->avail_out == 0 || ret == LZMA_STREAM_END) {
size_t write_size = sizeof(outbuf) - strm->avail_out;
if (fwrite(outbuf, 1, write_size, outfile)
!= write_size) {
fprintf(stderr, "Write error: %s\n",
strerror(errno));
return false;
}
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
}
if (ret != LZMA_OK) {
if (ret == LZMA_STREAM_END)
return true;
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_DATA_ERROR:
msg = "File size limits exceeded";
break;
default:
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Encoder error: %s (error code %u)\n",
msg, ret);
return false;
}
}
}
extern int
main(void)
{
lzma_stream strm = LZMA_STREAM_INIT;
bool success = init_encoder(&strm);
if (success)
success = compress(&strm, stdin, stdout);
lzma_end(&strm);
if (fclose(stdout)) {
fprintf(stderr, "Write error: %s\n", strerror(errno));
success = false;
}
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}

View file

@ -0,0 +1,205 @@
// SPDX-License-Identifier: 0BSD
///////////////////////////////////////////////////////////////////////////////
//
/// \file 04_compress_easy_mt.c
/// \brief Compress in multi-call mode using LZMA2 in multi-threaded mode
///
/// Usage: ./04_compress_easy_mt < INFILE > OUTFILE
///
/// Example: ./04_compress_easy_mt < foo > foo.xz
//
// Author: Lasse Collin
//
///////////////////////////////////////////////////////////////////////////////
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <lzma.h>
static bool
init_encoder(lzma_stream *strm)
{
// The threaded encoder takes the options as pointer to
// a lzma_mt structure.
lzma_mt mt = {
// No flags are needed.
.flags = 0,
// Let liblzma determine a sane block size.
.block_size = 0,
// Use no timeout for lzma_code() calls by setting timeout
// to zero. That is, sometimes lzma_code() might block for
// a long time (from several seconds to even minutes).
// If this is not OK, for example due to progress indicator
// needing updates, specify a timeout in milliseconds here.
// See the documentation of lzma_mt in lzma/container.h for
// information how to choose a reasonable timeout.
.timeout = 0,
// Use the default preset (6) for LZMA2.
// To use a preset, filters must be set to NULL.
.preset = LZMA_PRESET_DEFAULT,
.filters = NULL,
// Use CRC64 for integrity checking. See also
// 01_compress_easy.c about choosing the integrity check.
.check = LZMA_CHECK_CRC64,
};
// Detect how many threads the CPU supports.
mt.threads = lzma_cputhreads();
// If the number of CPU cores/threads cannot be detected,
// use one thread. Note that this isn't the same as the normal
// single-threaded mode as this will still split the data into
// blocks and use more RAM than the normal single-threaded mode.
// You may want to consider using lzma_easy_encoder() or
// lzma_stream_encoder() instead of lzma_stream_encoder_mt() if
// lzma_cputhreads() returns 0 or 1.
if (mt.threads == 0)
mt.threads = 1;
// If the number of CPU cores/threads exceeds threads_max,
// limit the number of threads to keep memory usage lower.
// The number 8 is arbitrarily chosen and may be too low or
// high depending on the compression preset and the computer
// being used.
//
// FIXME: A better way could be to check the amount of RAM
// (or available RAM) and use lzma_stream_encoder_mt_memusage()
// to determine if the number of threads should be reduced.
const uint32_t threads_max = 8;
if (mt.threads > threads_max)
mt.threads = threads_max;
// Initialize the threaded encoder.
lzma_ret ret = lzma_stream_encoder_mt(strm, &mt);
if (ret == LZMA_OK)
return true;
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_OPTIONS_ERROR:
// We are no longer using a plain preset so this error
// message has been edited accordingly compared to
// 01_compress_easy.c.
msg = "Specified filter chain is not supported";
break;
case LZMA_UNSUPPORTED_CHECK:
msg = "Specified integrity check is not supported";
break;
default:
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Error initializing the encoder: %s (error code %u)\n",
msg, ret);
return false;
}
// This function is identical to the one in 01_compress_easy.c.
static bool
compress(lzma_stream *strm, FILE *infile, FILE *outfile)
{
lzma_action action = LZMA_RUN;
uint8_t inbuf[BUFSIZ];
uint8_t outbuf[BUFSIZ];
strm->next_in = NULL;
strm->avail_in = 0;
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
while (true) {
if (strm->avail_in == 0 && !feof(infile)) {
strm->next_in = inbuf;
strm->avail_in = fread(inbuf, 1, sizeof(inbuf),
infile);
if (ferror(infile)) {
fprintf(stderr, "Read error: %s\n",
strerror(errno));
return false;
}
if (feof(infile))
action = LZMA_FINISH;
}
lzma_ret ret = lzma_code(strm, action);
if (strm->avail_out == 0 || ret == LZMA_STREAM_END) {
size_t write_size = sizeof(outbuf) - strm->avail_out;
if (fwrite(outbuf, 1, write_size, outfile)
!= write_size) {
fprintf(stderr, "Write error: %s\n",
strerror(errno));
return false;
}
strm->next_out = outbuf;
strm->avail_out = sizeof(outbuf);
}
if (ret != LZMA_OK) {
if (ret == LZMA_STREAM_END)
return true;
const char *msg;
switch (ret) {
case LZMA_MEM_ERROR:
msg = "Memory allocation failed";
break;
case LZMA_DATA_ERROR:
msg = "File size limits exceeded";
break;
default:
msg = "Unknown error, possibly a bug";
break;
}
fprintf(stderr, "Encoder error: %s (error code %u)\n",
msg, ret);
return false;
}
}
}
extern int
main(void)
{
lzma_stream strm = LZMA_STREAM_INIT;
bool success = init_encoder(&strm);
if (success)
success = compress(&strm, stdin, stdout);
lzma_end(&strm);
if (fclose(stdout)) {
fprintf(stderr, "Write error: %s\n", strerror(errno));
success = false;
}
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}

View file

@ -0,0 +1,205 @@
// SPDX-License-Identifier: 0BSD
///////////////////////////////////////////////////////////////////////////////
//
/// \file 11_file_info.c
/// \brief Get uncompressed size of .xz file(s)
///
/// Usage: ./11_file_info INFILE1.xz [INFILEn.xz]...
///
/// Example: ./11_file_info foo.xz
//
// Author: Lasse Collin
//
///////////////////////////////////////////////////////////////////////////////
#include <stdbool.h>
#include <inttypes.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <lzma.h>
static bool
print_file_size(lzma_stream *strm, FILE *infile, const char *filename)
{
// Get the file size. In standard C it can be done by seeking to
// the end of the file and then getting the file position.
// In POSIX one can use fstat() and then st_size from struct stat.
// Also note that fseek() and ftell() use long and thus don't support
// large files on 32-bit systems (POSIX versions fseeko() and
// ftello() can support large files).
if (fseek(infile, 0, SEEK_END)) {
fprintf(stderr, "Error seeking the file '%s': %s\n",
filename, strerror(errno));
return false;
}
const long file_size = ftell(infile);
// The decoder wants to start from the beginning of the .xz file.
rewind(infile);
// Initialize the decoder.
lzma_index *i;
lzma_ret ret = lzma_file_info_decoder(strm, &i, UINT64_MAX,
(uint64_t)file_size);
switch (ret) {
case LZMA_OK:
// Initialization succeeded.
break;
case LZMA_MEM_ERROR:
fprintf(stderr, "Out of memory when initializing "
"the .xz file info decoder\n");
return false;
case LZMA_PROG_ERROR:
default:
fprintf(stderr, "Unknown error, possibly a bug\n");
return false;
}
// This example program reuses the same lzma_stream structure
// for multiple files, so we need to reset this when starting
// a new file.
strm->avail_in = 0;
// Buffer for input data.
uint8_t inbuf[BUFSIZ];
// Pass data to the decoder and seek when needed.
while (true) {
if (strm->avail_in == 0) {
strm->next_in = inbuf;
strm->avail_in = fread(inbuf, 1, sizeof(inbuf),
infile);
if (ferror(infile)) {
fprintf(stderr,
"Error reading from '%s': %s\n",
filename, strerror(errno));
return false;
}
// We don't need to care about hitting the end of
// the file so no need to check for feof().
}
ret = lzma_code(strm, LZMA_RUN);
switch (ret) {
case LZMA_OK:
break;
case LZMA_SEEK_NEEDED:
// The cast is safe because liblzma won't ask us to
// seek past the known size of the input file which
// did fit into a long.
//
// NOTE: Remember to change these to off_t if you
// switch fseeko() or lseek().
if (fseek(infile, (long)(strm->seek_pos), SEEK_SET)) {
fprintf(stderr, "Error seeking the "
"file '%s': %s\n",
filename, strerror(errno));
return false;
}
// The old data in the inbuf is useless now. Set
// avail_in to zero so that we will read new input
// from the new file position on the next iteration
// of this loop.
strm->avail_in = 0;
break;
case LZMA_STREAM_END:
// File information was successfully decoded.
// See <lzma/index.h> for functions that can be
// used on it. In this example we just print
// the uncompressed size (in bytes) of
// the .xz file followed by its file name.
printf("%10" PRIu64 " %s\n",
lzma_index_uncompressed_size(i),
filename);
// Free the memory of the lzma_index structure.
lzma_index_end(i, NULL);
return true;
case LZMA_FORMAT_ERROR:
// .xz magic bytes weren't found.
fprintf(stderr, "The file '%s' is not "
"in the .xz format\n", filename);
return false;
case LZMA_OPTIONS_ERROR:
fprintf(stderr, "The file '%s' has .xz headers that "
"are not supported by this liblzma "
"version\n", filename);
return false;
case LZMA_DATA_ERROR:
fprintf(stderr, "The file '%s' is corrupt\n",
filename);
return false;
case LZMA_MEM_ERROR:
fprintf(stderr, "Memory allocation failed when "
"decoding the file '%s'\n", filename);
return false;
// LZMA_MEMLIMIT_ERROR shouldn't happen because we used
// UINT64_MAX as the limit.
//
// LZMA_BUF_ERROR shouldn't happen because we always provide
// new input when the input buffer is empty. The decoder
// knows the input file size and thus won't try to read past
// the end of the file.
case LZMA_MEMLIMIT_ERROR:
case LZMA_BUF_ERROR:
case LZMA_PROG_ERROR:
default:
fprintf(stderr, "Unknown error, possibly a bug\n");
return false;
}
}
// This line is never reached.
}
extern int
main(int argc, char **argv)
{
bool success = true;
lzma_stream strm = LZMA_STREAM_INIT;
for (int i = 1; i < argc; ++i) {
FILE *infile = fopen(argv[i], "rb");
if (infile == NULL) {
fprintf(stderr, "Cannot open the file '%s': %s\n",
argv[i], strerror(errno));
success = false;
}
success &= print_file_size(&strm, infile, argv[i]);
(void)fclose(infile);
}
lzma_end(&strm);
// Close stdout to catch possible write errors that can occur
// when pending data is flushed from the stdio buffers.
if (fclose(stdout)) {
fprintf(stderr, "Write error: %s\n", strerror(errno));
success = false;
}
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}

View file

@ -0,0 +1,21 @@
# SPDX-License-Identifier: 0BSD
# Author: Lasse Collin
CC = c99
CFLAGS = -g
LDFLAGS = -llzma
PROGS = \
01_compress_easy \
02_decompress \
03_compress_custom \
04_compress_easy_mt \
11_file_info
all: $(PROGS)
.c:
$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)
clean:
-rm -f $(PROGS)

View file

@ -0,0 +1,244 @@
XZ Utils FAQ
============
Q: What do the letters XZ mean?
A: Nothing. They are just two letters, which come from the file format
suffix .xz. The .xz suffix was selected, because it seemed to be
pretty much unused. It has no deeper meaning.
Q: What are LZMA and LZMA2?
A: LZMA stands for Lempel-Ziv-Markov chain-Algorithm. It is the name
of the compression algorithm designed by Igor Pavlov for 7-Zip.
LZMA is based on LZ77 and range encoding.
LZMA2 is an updated version of the original LZMA to fix a couple of
practical issues. In context of XZ Utils, LZMA is called LZMA1 to
emphasize that LZMA is not the same thing as LZMA2. LZMA2 is the
primary compression algorithm in the .xz file format.
Q: There are many LZMA related projects. How does XZ Utils relate to them?
A: 7-Zip and LZMA SDK are the original projects. LZMA SDK is roughly
a subset of the 7-Zip source tree.
p7zip is 7-Zip's command-line tools ported to POSIX-like systems.
LZMA Utils provide a gzip-like lzma tool for POSIX-like systems.
LZMA Utils are based on LZMA SDK. XZ Utils are the successor to
LZMA Utils.
There are several other projects using LZMA. Most are more or less
based on LZMA SDK. See <https://7-zip.org/links.html>.
Q: Why is liblzma named liblzma if its primary file format is .xz?
Shouldn't it be e.g. libxz?
A: When the designing of the .xz format began, the idea was to replace
the .lzma format and use the same .lzma suffix. It would have been
quite OK to reuse the suffix when there were very few .lzma files
around. However, the old .lzma format became popular before the
new format was finished. The new format was renamed to .xz but the
name of liblzma wasn't changed.
Q: Do XZ Utils support the .7z format?
A: No. Use 7-Zip (Windows) or p7zip (POSIX-like systems) to handle .7z
files.
Q: I have many .tar.7z files. Can I convert them to .tar.xz without
spending hours recompressing the data?
A: In the "extra" directory, there is a script named 7z2lzma.bash which
is able to convert some .7z files to the .lzma format (not .xz). It
needs the 7za (or 7z) command from p7zip. The script may silently
produce corrupt output if certain assumptions are not met, so
decompress the resulting .lzma file and compare it against the
original before deleting the original file!
Q: I have many .lzma files. Can I quickly convert them to the .xz format?
A: For now, no. Since XZ Utils supports the .lzma format, it's usually
not too bad to keep the old files in the old format. If you want to
do the conversion anyway, you need to decompress the .lzma files and
then recompress to the .xz format.
Technically, there is a way to make the conversion relatively fast
(roughly twice the time that normal decompression takes). Writing
such a tool would take quite a bit of time though, and would probably
be useful to only a few people. If you really want such a conversion
tool, contact Lasse Collin and offer some money.
Q: I have installed xz, but my tar doesn't recognize .tar.xz files.
How can I extract .tar.xz files?
A: xz -dc foo.tar.xz | tar xf -
Q: Can I recover parts of a broken .xz file (e.g. a corrupted CD-R)?
A: It may be possible if the file consists of multiple blocks, which
typically is not the case if the file was created in single-threaded
mode. There is no recovery program yet.
Q: Is (some part of) XZ Utils patented?
A: Lasse Collin is not aware of any patents that could affect XZ Utils.
However, due to the nature of software patents, it's not possible to
guarantee that XZ Utils isn't affected by any third party patent(s).
Q: Where can I find documentation about the file format and algorithms?
A: The .xz format is documented in xz-file-format.txt. It is a container
format only, and doesn't include descriptions of any non-trivial
filters.
Documenting LZMA and LZMA2 is planned, but for now, there is no other
documentation than the source code. Before you begin, you should know
the basics of LZ77 and range-coding algorithms. LZMA is based on LZ77,
but LZMA is a lot more complex. Range coding is used to compress
the final bitstream like Huffman coding is used in Deflate.
Q: I cannot find BCJ and BCJ2 filters. Don't they exist in liblzma?
A: BCJ filter is called "x86" in liblzma. BCJ2 is not included,
because it requires using more than one encoded output stream.
Q: I need to use a script that runs "xz -9". On a system with 256 MiB
of RAM, xz says that it cannot allocate memory. Can I make the
script work without modifying it?
A: Set a default memory usage limit for compression. You can do it e.g.
in a shell initialization script such as ~/.bashrc or /etc/profile:
XZ_DEFAULTS=--memlimit-compress=150MiB
export XZ_DEFAULTS
xz will then scale the compression settings down so that the given
memory usage limit is not reached. This way xz shouldn't run out
of memory.
Check also that memory-related resource limits are high enough.
On most systems, "ulimit -a" will show the current resource limits.
Q: How do I create files that can be decompressed with XZ Embedded?
A: See the documentation in XZ Embedded. In short, something like
this is a good start:
xz --check=crc32 --lzma2=preset=6e,dict=64KiB
Or if a BCJ filter is needed too, e.g. if compressing
a kernel image for PowerPC:
xz --check=crc32 --powerpc --lzma2=preset=6e,dict=64KiB
Adjust the dictionary size to get a good compromise between
compression ratio and decompressor memory usage. Note that
in single-call decompression mode of XZ Embedded, a big
dictionary doesn't increase memory usage.
Q: How is multi-threaded compression implemented in XZ Utils?
A: The simplest method is splitting the uncompressed data into blocks
and compressing them in parallel independent from each other.
This is currently the only threading method supported in XZ Utils.
Since the blocks are compressed independently, they can also be
decompressed independently. Together with the index feature in .xz,
this allows using threads to create .xz files for random-access
reading. This also makes threaded decompression possible.
The independent blocks method has a couple of disadvantages too. It
will compress worse than a single-block method. Often the difference
is not too big (maybe 1-2 %) but sometimes it can be too big. Also,
the memory usage of the compressor increases linearly when adding
threads.
At least two other threading methods are possible but these haven't
been implemented in XZ Utils:
Match finder parallelization has been in 7-Zip for ages. It doesn't
affect compression ratio or memory usage significantly. Among the
three threading methods, only this is useful when compressing small
files (files that are not significantly bigger than the dictionary).
Unfortunately this method scales only to about two CPU cores.
The third method is pigz-style threading (I use that name, because
pigz <https://www.zlib.net/pigz/> uses that method). It doesn't
affect compression ratio significantly and scales to many cores.
The memory usage scales linearly when threads are added. This isn't
significant with pigz, because Deflate uses only a 32 KiB dictionary,
but with LZMA2 the memory usage will increase dramatically just like
with the independent-blocks method. There is also a constant
computational overhead, which may make pigz-method a bit dull on
dual-core compared to the parallel match finder method, but with more
cores the overhead is not a big deal anymore.
Combining the threading methods will be possible and also useful.
For example, combining match finder parallelization with pigz-style
threading or independent-blocks-threading can cut the memory usage
by 50 %.
Q: I told xz to use many threads but it is using only one or two
processor cores. What is wrong?
A: Since multi-threaded compression is done by splitting the data into
blocks that are compressed individually, if the input file is too
small for the block size, then many threads cannot be used. The
default block size increases when the compression level is
increased. For example, xz -6 uses 8 MiB LZMA2 dictionary and
24 MiB blocks, and xz -9 uses 64 MiB LZMA dictionary and 192 MiB
blocks. If the input file is 100 MiB, xz -6 can use five threads
of which one will finish quickly as it has only 4 MiB to compress.
However, for the same file, xz -9 can only use one thread.
One can adjust block size with --block-size=SIZE but making the
block size smaller than LZMA2 dictionary is waste of RAM: using
xz -9 with 6 MiB blocks isn't any better than using xz -6 with
6 MiB blocks. The default settings use a block size bigger than
the LZMA2 dictionary size because this was seen as a reasonable
compromise between RAM usage and compression ratio.
When decompressing, the ability to use threads depends on how the
file was created. If it was created in multi-threaded mode then
it can be decompressed in multi-threaded mode too if there are
multiple blocks in the file.
Q: How do I build a program that needs liblzmadec (lzmadec.h)?
A: liblzmadec is part of LZMA Utils. XZ Utils has liblzma, but no
liblzmadec. The code using liblzmadec should be ported to use
liblzma instead. If you cannot or don't want to do that, download
LZMA Utils from <https://tukaani.org/lzma/>.
Q: The default build of liblzma is too big. How can I make it smaller?
A: Give --enable-small to the configure script. Use also appropriate
--enable or --disable options to include only those filter encoders
and decoders and integrity checks that you actually need. Use
CFLAGS=-Os (with GCC) or equivalent to tell your compiler to optimize
for size. See INSTALL for information about configure options.
If the result is still too big, take a look at XZ Embedded. It is
a separate project, which provides a limited but significantly
smaller XZ decoder implementation than XZ Utils. You can find it
at <https://tukaani.org/xz/embedded.html>.

View file

@ -0,0 +1,150 @@
History of LZMA Utils and XZ Utils
==================================
Tukaani distribution
In 2005, there was a small group working on the Tukaani distribution,
which was a Slackware fork. One of the project's goals was to fit the
distro on a single 700 MiB ISO-9660 image. Using LZMA instead of gzip
helped a lot. Roughly speaking, one could fit data that took 1000 MiB
in gzipped form into 700 MiB with LZMA. Naturally, the compression
ratio varied across packages, but this was what we got on average.
Slackware packages have traditionally had .tgz as the filename suffix,
which is an abbreviation of .tar.gz. A logical naming for LZMA
compressed packages was .tlz, being an abbreviation of .tar.lzma.
At the end of the year 2007, there was no distribution under the
Tukaani project anymore, but development of LZMA Utils was kept going.
Still, there were .tlz packages around, because at least Vector Linux
(a Slackware based distribution) used LZMA for its packages.
First versions of the modified pkgtools used the LZMA_Alone tool from
Igor Pavlov's LZMA SDK as is. It was fine, because users wouldn't need
to interact with LZMA_Alone directly. But people soon wanted to use
LZMA for other files too, and the interface of LZMA_Alone wasn't
comfortable for those used to gzip and bzip2.
First steps of LZMA Utils
The first version of LZMA Utils (4.22.0) included a shell script called
lzmash. It was a wrapper that had a gzip-like command-line interface. It
used the LZMA_Alone tool from LZMA SDK to do all the real work. zgrep,
zdiff, and related scripts from gzip were adapted to work with LZMA and
were part of the first LZMA Utils release too.
LZMA Utils 4.22.0 included also lzmadec, which was a small (less than
10 KiB) decoder-only command-line tool. It was written on top of the
decoder-only C code found from the LZMA SDK. lzmadec was convenient in
situations where LZMA_Alone (a few hundred KiB) would be too big.
lzmash and lzmadec were written by Lasse Collin.
Second generation
The lzmash script was an ugly and not very secure hack. The last
version of LZMA Utils to use lzmash was 4.27.1.
LZMA Utils 4.32.0beta1 introduced a new lzma command-line tool written
by Ville Koskinen. It was written in C++, and used the encoder and
decoder from C++ LZMA SDK with some little modifications. This tool
replaced both the lzmash script and the LZMA_Alone command-line tool
in LZMA Utils.
Introducing this new tool caused some temporary incompatibilities,
because the LZMA_Alone executable was simply named lzma like the new
command-line tool, but they had a completely different command-line
interface. The file format was still the same.
Lasse wrote liblzmadec, which was a small decoder-only library based
on the C code found from LZMA SDK. liblzmadec had an API similar to
zlib, although there were some significant differences, which made it
non-trivial to use it in some applications designed for zlib and
libbzip2.
The lzmadec command-line tool was converted to use liblzmadec.
Alexandre Sauvé helped converting the build system to use GNU
Autotools. This made it easier to test for certain less portable
features needed by the new command-line tool.
Since the new command-line tool never got completely finished (for
example, it didn't support the LZMA_OPT environment variable), the
intent was to not call 4.32.x stable. Similarly, liblzmadec wasn't
polished, but appeared to work well enough, so some people started
using it too.
Because the development of the third generation of LZMA Utils was
delayed considerably (3-4 years), the 4.32.x branch had to be kept
maintained. It got some bug fixes now and then, and finally it was
decided to call it stable, although most of the missing features were
never added.
File format problems
The file format used by LZMA_Alone was primitive. It was designed with
embedded systems in mind, and thus provided only a minimal set of
features. The two biggest problems for non-embedded use were the lack
of magic bytes and an integrity check.
Igor and Lasse started developing a new file format with some help
from Ville Koskinen. Also Mark Adler, Mikko Pouru, H. Peter Anvin,
and Lars Wirzenius helped with some minor things at some point of the
development. Designing the new format took quite a long time (actually,
too long a time would be a more appropriate expression). It was mostly
because Lasse was quite slow at getting things done due to personal
reasons.
Originally the new format was supposed to use the same .lzma suffix
that was already used by the old file format. Switching to the new
format wouldn't have caused much trouble when the old format wasn't
used by many people. But since the development of the new format took
such a long time, the old format got quite popular, and it was decided
that the new file format must use a different suffix.
It was decided to use .xz as the suffix of the new file format. The
first stable .xz file format specification was finally released in
December 2008. In addition to fixing the most obvious problems of
the old .lzma format, the .xz format added some new features like
support for multiple filters (compression algorithms), filter chaining
(like piping on the command line), and limited random-access reading.
Currently the primary compression algorithm used in .xz is LZMA2.
It is an extension on top of the original LZMA to fix some practical
problems: LZMA2 adds support for flushing the encoder, uncompressed
chunks, eases stateful decoder implementations, and improves support
for multithreading. Since LZMA2 is better than the original LZMA, the
original LZMA is not supported in .xz.
Transition to XZ Utils
The early versions of XZ Utils were called LZMA Utils. The first
releases were 4.42.0alphas. They dropped the rest of the C++ LZMA SDK.
The code was still directly based on LZMA SDK but ported to C and
converted from a callback API to a stateful API. Later, Igor Pavlov
made a C version of the LZMA encoder too; these ports from C++ to C
were independent in LZMA SDK and LZMA Utils.
The core of the new LZMA Utils was liblzma, a compression library with
a zlib-like API. liblzma supported both the old and new file format.
The gzip-like lzma command-line tool was rewritten to use liblzma.
The new LZMA Utils code base was renamed to XZ Utils when the name
of the new file format had been decided. The liblzma compression
library retained its name though, because changing it would have
caused unnecessary breakage in applications already using the early
liblzma snapshots.
The xz command-line tool can emulate the gzip-like lzma tool by
creating appropriate symlinks (e.g. lzma -> xz). Thus, practically
all scripts using the lzma tool from LZMA Utils will work as is with
XZ Utils (and will keep using the old .lzma format). Still, the .lzma
format is more or less deprecated. XZ Utils will keep supporting it,
but new applications should use the .xz format, and migrating old
applications to .xz is often a good idea too.

View file

@ -0,0 +1,115 @@
EXPORTS
lzma_alone_decoder
lzma_alone_encoder
lzma_auto_decoder
lzma_bcj_arm64_decode
lzma_bcj_arm64_encode
lzma_bcj_riscv_decode
lzma_bcj_riscv_encode
lzma_bcj_x86_decode
lzma_bcj_x86_encode
lzma_block_buffer_bound
lzma_block_buffer_decode
lzma_block_buffer_encode
lzma_block_compressed_size
lzma_block_decoder
lzma_block_encoder
lzma_block_header_decode
lzma_block_header_encode
lzma_block_header_size
lzma_block_total_size
lzma_block_uncomp_encode
lzma_block_unpadded_size
lzma_check_is_supported
lzma_check_size
lzma_code
lzma_cputhreads
lzma_crc32
lzma_crc64
lzma_easy_buffer_encode
lzma_easy_decoder_memusage
lzma_easy_encoder
lzma_easy_encoder_memusage
lzma_end
lzma_file_info_decoder
lzma_filter_decoder_is_supported
lzma_filter_encoder_is_supported
lzma_filter_flags_decode
lzma_filter_flags_encode
lzma_filter_flags_size
lzma_filters_copy
lzma_filters_free
lzma_filters_update
lzma_get_check
lzma_get_progress
lzma_index_append
lzma_index_block_count
lzma_index_buffer_decode
lzma_index_buffer_encode
lzma_index_cat
lzma_index_checks
lzma_index_decoder
lzma_index_dup
lzma_index_encoder
lzma_index_end
lzma_index_file_size
lzma_index_hash_append
lzma_index_hash_decode
lzma_index_hash_end
lzma_index_hash_init
lzma_index_hash_size
lzma_index_init
lzma_index_iter_init
lzma_index_iter_locate
lzma_index_iter_next
lzma_index_iter_rewind
lzma_index_memusage
lzma_index_memused
lzma_index_size
lzma_index_stream_count
lzma_index_stream_flags
lzma_index_stream_padding
lzma_index_stream_size
lzma_index_total_size
lzma_index_uncompressed_size
lzma_lzip_decoder
lzma_lzma_preset
lzma_memlimit_get
lzma_memlimit_set
lzma_memusage
lzma_mf_is_supported
lzma_microlzma_decoder
lzma_microlzma_encoder
lzma_mode_is_supported
lzma_mt_block_size
lzma_physmem
lzma_properties_decode
lzma_properties_encode
lzma_properties_size
lzma_raw_buffer_decode
lzma_raw_buffer_encode
lzma_raw_decoder
lzma_raw_decoder_memusage
lzma_raw_encoder
lzma_raw_encoder_memusage
lzma_str_from_filters
lzma_str_list_filters
lzma_str_to_filters
lzma_stream_buffer_bound
lzma_stream_buffer_decode
lzma_stream_buffer_encode
lzma_stream_decoder
lzma_stream_decoder_mt
lzma_stream_encoder
lzma_stream_encoder_mt
lzma_stream_encoder_mt_memusage
lzma_stream_flags_compare
lzma_stream_footer_decode
lzma_stream_footer_encode
lzma_stream_header_decode
lzma_stream_header_encode
lzma_version_number
lzma_version_string
lzma_vli_decode
lzma_vli_encode
lzma_vli_size

View file

@ -0,0 +1,173 @@
The .lzma File Format
=====================
0. Preface
0.1. Notices and Acknowledgements
0.2. Changes
1. File Format
1.1. Header
1.1.1. Properties
1.1.2. Dictionary Size
1.1.3. Uncompressed Size
1.2. LZMA Compressed Data
2. References
0. Preface
This document describes the .lzma file format, which is
sometimes also called LZMA_Alone format. It is a legacy file
format, which is being or has been replaced by the .xz format.
The MIME type of the .lzma format is `application/x-lzma'.
The most commonly used software to handle .lzma files are
LZMA SDK, LZMA Utils, 7-Zip, and XZ Utils. This document
describes some of the differences between these implementations
and gives hints what subset of the .lzma format is the most
portable.
0.1. Notices and Acknowledgements
This file format was designed by Igor Pavlov for use in
LZMA SDK. This document was written by Lasse Collin
<lasse.collin@tukaani.org> using the documentation found
from the LZMA SDK.
This document has been put into the public domain.
0.2. Changes
Last modified: 2024-04-08 17:35+0300
From version 2011-04-12 11:55+0300 to 2022-07-13 21:00+0300:
The section 1.1.3 was modified to allow End of Payload Marker
with a known Uncompressed Size.
1. File Format
+-+-+-+-+-+-+-+-+-+-+-+-+-+==========================+
| Header | LZMA Compressed Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+==========================+
The .lzma format file consist of 13-byte Header followed by
the LZMA Compressed Data.
Unlike the .gz, .bz2, and .xz formats, it is not possible to
concatenate multiple .lzma files as is and expect the
decompression tool to decode the resulting file as if it were
a single .lzma file.
For example, the command line tools from LZMA Utils and
LZMA SDK silently ignore all the data after the first .lzma
stream. In contrast, the command line tool from XZ Utils
considers the .lzma file to be corrupt if there is data after
the first .lzma stream.
1.1. Header
+------------+----+----+----+----+--+--+--+--+--+--+--+--+
| Properties | Dictionary Size | Uncompressed Size |
+------------+----+----+----+----+--+--+--+--+--+--+--+--+
1.1.1. Properties
The Properties field contains three properties. An abbreviation
is given in parentheses, followed by the value range of the
property. The field consists of
1) the number of literal context bits (lc, [0, 8]);
2) the number of literal position bits (lp, [0, 4]); and
3) the number of position bits (pb, [0, 4]).
The properties are encoded using the following formula:
Properties = (pb * 5 + lp) * 9 + lc
The following C code illustrates a straightforward way to
decode the Properties field:
uint8_t lc, lp, pb;
uint8_t prop = get_lzma_properties();
if (prop > (4 * 5 + 4) * 9 + 8)
return LZMA_PROPERTIES_ERROR;
pb = prop / (9 * 5);
prop -= pb * 9 * 5;
lp = prop / 9;
lc = prop - lp * 9;
XZ Utils has an additional requirement: lc + lp <= 4. Files
which don't follow this requirement cannot be decompressed
with XZ Utils. Usually this isn't a problem since the most
common lc/lp/pb values are 3/0/2. It is the only lc/lp/pb
combination that the files created by LZMA Utils can have,
but LZMA Utils can decompress files with any lc/lp/pb.
1.1.2. Dictionary Size
Dictionary Size is stored as an unsigned 32-bit little endian
integer. Any 32-bit value is possible, but for maximum
portability, only sizes of 2^n and 2^n + 2^(n-1) should be
used.
LZMA Utils creates only files with dictionary size 2^n,
16 <= n <= 25. LZMA Utils can decompress files with any
dictionary size.
XZ Utils creates and decompresses .lzma files only with
dictionary sizes 2^n and 2^n + 2^(n-1). If some other
dictionary size is specified when compressing, the value
stored in the Dictionary Size field is a rounded up, but the
specified value is still used in the actual compression code.
1.1.3. Uncompressed Size
Uncompressed Size is stored as unsigned 64-bit little endian
integer. A special value of 0xFFFF_FFFF_FFFF_FFFF indicates
that Uncompressed Size is unknown. End of Payload Marker (*)
is used if Uncompressed Size is unknown. End of Payload Marker
is allowed but rarely used if Uncompressed Size is known.
XZ Utils 5.2.5 and older don't support .lzma files that have
End of Payload Marker together with a known Uncompressed Size.
XZ Utils rejects files whose Uncompressed Size field specifies
a known size that is 256 GiB or more. This is to reject false
positives when trying to guess if the input file is in the
.lzma format. When Uncompressed Size is unknown, there is no
limit for the uncompressed size of the file.
(*) Some tools use the term End of Stream (EOS) marker
instead of End of Payload Marker.
1.2. LZMA Compressed Data
Detailed description of the format of this field is out of
scope of this document.
2. References
LZMA SDK - The original LZMA implementation
https://7-zip.org/sdk.html
7-Zip
https://7-zip.org/
LZMA Utils - LZMA adapted to POSIX-like systems
https://tukaani.org/lzma/
XZ Utils - The next generation of LZMA Utils
https://tukaani.org/xz/
The .xz file format - The successor of the .lzma format
https://tukaani.org/xz/xz-file-format.txt

File diff suppressed because it is too large Load diff