What if you spent thousands of dollars building a gorgeous law firm website, carefully wrote content about each of your practice areas, and then accidentally put up a "Do Not Enter" sign that kept Google from seeing any of it? It sounds absurd, but it happens more often than you'd believe. A small file called robots.txt could be silently telling Google to stay away from your most important pages — and you'd never know unless you checked.
What robots.txt actually means
Every website can have a small text file called robots.txt that gives instructions to search engines about which parts of the site they're allowed to visit and which parts they should ignore.
Think of it as a set of signs at the entrance to a building. Some doors say "Welcome, come on in." Others say "Staff only — do not enter." The robots.txt file works the same way for search engine crawlers (the automated programs Google sends to read and catalog your website).
There are legitimate reasons to block search engines from certain pages. You probably don't want Google indexing your website's admin login page, test pages, or internal documents. The robots.txt file is how you communicate those boundaries.
The problem arises when the file is misconfigured — when it accidentally blocks pages you do want Google to find. And unlike a physical "Do Not Enter" sign that you'd walk past every day and notice, robots.txt sits quietly on your server where most law firm owners never look.
Why this matters for your law firm
A misconfigured robots.txt file can cause real damage to your online visibility:
- Entire sections of your site can disappear from Google. If your robots.txt accidentally blocks your practice area pages, blog posts, or attorney profiles, Google won't index them. It's as if those pages don't exist as far as search results are concerned.
- It can happen without warning. There are no alerts. Your website still works perfectly for anyone who types in the address directly. Everything looks fine from the inside — but Google is being turned away at the door.
- It's a common side effect of website migrations. One of the most frequent causes of robots.txt problems is when a law firm moves to a new website or redesigns their existing one. During development, web teams often block search engines so the unfinished site doesn't get indexed. Sometimes they forget to remove that block when the site goes live.
- Even partial blocks can hurt. Maybe Google can see your homepage but is blocked from your blog or your individual practice area pages. You'd still show up in some searches but miss out on dozens of others.
How to check if your site has this
Checking your robots.txt file is straightforward:
- Open your browser and type your website address followed by /robots.txt. For example: www.yourfirm.com/robots.txt
- A text file will load. It's short and simple — usually just a few lines.
- Look for any lines that say "Disallow:" followed by a forward slash. Here's what to watch for:
- Disallow: / — This is the big red flag. A single forward slash after "Disallow" means the entire site is blocked. Google cannot access anything. This is the most common and most damaging mistake.
- Disallow: /blog/ or Disallow: /practice-areas/ — This means specific sections are blocked. If these are pages you want appearing in Google, there's a problem.
- Disallow: (with nothing after it) — This is actually fine. An empty disallow means nothing is blocked.
If you have access to Google Search Console, there's also a robots.txt testing tool that will tell you whether specific pages on your site are blocked or accessible.
What to do next
If you checked your robots.txt and everything looks clean — no important sections are blocked — you're in good shape. But if you found something concerning, here's what to do:
- Don't edit the file yourself unless you're comfortable with how it works. A well-meaning but incorrect change could make things worse. Instead, share what you found with your web developer and ask them to review it.
- If you see "Disallow: /", contact your web team immediately. This is urgent — your entire site is invisible to Google and you're getting zero organic search traffic.
- Ask your developer to review it after any website migration or redesign. Make it part of your launch checklist. A two-minute review of robots.txt can prevent months of lost search visibility.
- Questions to ask your web team: "Can you confirm our robots.txt file isn't blocking any important pages?" and "Is our robots.txt set up correctly for SEO?"
This is one of the first things the LexGrow SEO platform checks during a visibility audit, because it's so common and so impactful. A single misconfigured line can undo months of SEO work.
The good news? Once robots.txt is fixed, the recovery is usually fast. Google will revisit your site, find the pages that were previously blocked, and start indexing them — often within days. So take 30 seconds right now, check your robots.txt, and make sure Google's welcome mat is out for every page that matters. It might be the most valuable half-minute you spend this week.