Wget is a great tool that allows you to crawl websites.

Suppose you had a webpage with 100 links to PDF files that you must download for one of the courses you are taking. Like this one.

It is too cumbersome to download them one by one manually, so you decide to use some wget magic.

$ wget -r -l 1 -A pdf -nd http://homepages.engineering.auckland.ac.nz/\~pkel015/SolidMechanicsBooks/Part_III/index.html

The flags mean

  • -r : recursive (follow links and download those too)
  • -l 1 : the level of recursion is 1: only download links from the current page
  • -A pdf : filter out extensions that are not in this list
  • -nd : no directory: do not create a separate directory, download the files them here

Bunch of output, then voilà. You have all the files:

$ ls
AnswersToSelectedProblems_Chapt01.pdf
AnswersToSelectedProblems_Chapt02.pdf
AnswersToSelectedProblems_Chapt04.pdf
Kinematics_of_CM_00.pdf
Kinematics_of_CM_01_Motion.pdf
Kinematics_of_CM_02_Deformation_Strain.pdf
Kinematics_of_CM_03_Deformation_Strain_Further.pdf
Kinematics_of_CM_04_Material_Time_Derivatives.pdf
Kinematics_of_CM_05_Deformation_Rates.pdf
Kinematics_of_CM_06_Deformation_Rates_Further.pdf
Kinematics_of_CM_07_Small_Strain_Theory.pdf
Kinematics_of_CM_08_Objectivity.pdf
Kinematics_of_CM_09_Rigid_Body_Rotations.pdf
Kinematics_of_CM_10_Convected_Coordinates.pdf
Kinematics_of_CM_11_Convected_Coordinates_Rates.pdf
Kinematics_of_CM_12_Pull_Back_Lie_Derivative.pdf
Kinematics_of_CM_13_Variation_Linearisation.pdf
References.pdf
robots.txt.tmp
Stress_Balance_Principles_00.pdf
Stress_Balance_Principles_01_Conservation_of_Mass.pdf
Stress_Balance_Principles_02_The_Momentum_Principles.pdf
Stress_Balance_Principles_03_The_Cauchy_Stress_Tensor.pdf
Stress_Balance_Principles_04_Properties_of_the_Stress_Tensor.pdf
Stress_Balance_Principles_05_Stress_Measures_NonLinear.pdf
Stress_Balance_Principles_06_Equations_of_Motion.pdf
Stress_Balance_Principles_07_Boundary_Problem.pdf
Stress_Balance_Principles_08_Cons_of_Mech_Energy.pdf
Stress_Balance_Principles_09_Virtual_Work.pdf
Stress_Balance_Principles_10_Convected_Coordinates.pdf
Thermomech_00.pdf
Thermomech_01_Classical_Thermo_First.pdf
Thermomech_02_Classical_Thermo_Second.pdf
Thermomech_03_Thermo_Functions.pdf
Thermomech_04_Continuum_Thermomechanics.pdf
Vectors_Tensors_00_Intro.pdf
Vectors_Tensors_01_Vector_Algebra.pdf
Vectors_Tensors_02_Vector_Space.pdf
Vectors_Tensors_03_Cartesian_Vectors.pdf
Vectors_Tensors_04_Matrices_and_Index_Notation.pdf
Vectors_Tensors_05_Coordinate_Transformation_Vectors.pdf
Vectors_Tensors_06_Vector_Calculus_1_Differentiation.pdf
Vectors_Tensors_07_Vector_Calculus_2_Integration.pdf
Vectors_Tensors_08_Tensors.pdf
Vectors_Tensors_09_Cartesian_Tensors.pdf
Vectors_Tensors_10_Special_Tensors.pdf
Vectors_Tensors_11_Eigenvalue.pdf
Vectors_Tensors_12_HigherOrderTensors.pdf
Vectors_Tensors_13_Coordinate_Transformation_Tensors.pdf
Vectors_Tensors_14_Tensor_Calculus.pdf
Vectors_Tensors_15_Tensor_Calculus_2.pdf
Vectors_Tensors_16_Curvilinear_Coordinates.pdf
Vectors_Tensors_17_Curvilinear_Transform.pdf
Vectors_Tensors_18_Curvilinear_Calculus.pdf
Vectors_Tensors_19_Curved_Geometries.pdf
Vectors_Tensors_20_Appendix_A.pdf
Vectors_Tensors_21_Appendix_B.pdf